Skill: newageinvestments25-byte/nai-opportunity-scout (version 1.0.0)

--- name: opportunity-scout description: "Hunt for real, expressed user pain points and unmet demand across Reddit, HN, and configurable sources. Finds demand signals like frustration posts, feature requests, workaround descriptions, and willingness-to-pay sentiment. Compiles findings into a prioritized digest with signal strength scoring, competition analysis, and trend tracking. Use when: opportunity scan, find opportunities, business ideas, market gaps, what are people asking for, niche research, demand signals, pain points, product ideas, gap analysis, what should I build, unmet needs." --- # Opportunity Scout Hunt for real demand signals — not news, not trends, but people expressing pain, frustration, and unmet needs that represent building opportunities. ## Skill Directory All paths below are relative to this skill's directory. - `scripts/configure.py` — manage niches, keywords, sources, schedule - `scripts/scan_sources.py` — generate search queries and process results - `scripts/score_signals.py` — score and rank findings - `scripts/digest.py` — generate prioritized markdown digest - `scripts/history.py` — track signals over time, detect trends - `references/signal-types.md` — what counts as a demand signal (read when scoring) - `references/source-guide.md` — how to configure sources effectively - `assets/config.example.json` — example niche configurations ## Data Files All state lives in the skill directory: - `config.json` — active configuration (created by configure.py) - `history.json` — signal history log (created by history.py) - `findings/` — raw and scored finding files per scan ## Workflow ### First-Time Setup 1. Run `configure.py --init` to create config.json from the example, or: - `configure.py --add-niche "AI tools for small business" --keywords "wish,need,looking for,alternative to,frustrated"` - `configure.py --add-source reddit:r/SaaS,reddit:r/smallbusiness,hackernews` - `configure.py --set-schedule daily` ### Running a Scan Execute these steps in order: 1. **Generate queries**: Run `scan_sources.py --generate-queries` to get optimized search queries. It prints JSON with query strings. 2. **Execute searches**: For each query, call the `web_search` tool. Collect all results into a JSON array and save to a temp file. 3. **Ingest results**: Run `scan_sources.py --ingest <results.json>` to parse raw search results into standardized findings. Outputs findings JSON. 4. **Score findings**: Run `score_signals.py <findings.json>` to score each finding on signal strength, engagement, freshness, competition, and recurrence. Outputs scored JSON. 5. **Update history**: Run `history.py --update <scored.json>` to log findings and detect trend patterns (persistent, emerging, fading). 6. **Generate digest**: Run `digest.py <scored.json>` to produce the markdown report. Use `--output <path>` to save to a specific location (e.g., Obsidian vault). Use `--max-results 20` to limit output. ### Quick Scan (Single Command Summary) For a rapid scan of a single niche without full config: 1. Run `scan_sources.py --quick "developer tools for AI agents"` to get queries 2. Execute web_search for each query 3. Pipe results through score and digest ### Reading References - Before scoring or evaluating signals manually, read `references/signal-types.md` for the taxonomy of demand signals and how to distinguish real demand from noise. - When helping users configure sources, read `references/source-guide.md`. ## Cron Integration Set schedule in config.json via `configure.py --set-schedule daily|weekly`. When triggered by cron, run the full scan workflow above. Save digest to the user's preferred output location (default: skill directory `findings/`). ## Key Design Principles - **Demand, not news**: Every finding should express unmet need, frustration, or a gap. Filter aggressively — 10 strong signals beat 100 weak ones. - **Batch queries**: Combine niche + keywords into fewer, broader queries rather than one query per keyword. Respect rate limits. - **Track over time**: Signals that persist across scans are more valuable than one-offs. Use history.py to surface persistent demand and fading trends. - **Score honestly**: High engagement + low competition + recurring = strong opportunity. Don't inflate scores — the user needs signal, not noise.

path: assets/config.example.json
content: {
  "version": 1,
  "schedule": "weekly",
  "default_signal_keywords": [
    "I wish",
    "someone should build",
    "looking for",
    "alternative to",
    "frustrated",
    "anyone solved",
    "workaround",
    "why is there no",
    "I'd pay for",
    "shut up and take my money",
    "hate that",
    "need a tool",
    "has anyone found",
    "does anyone know of"
  ],
  "niches": [
    {
      "name": "AI tools for small business",
      "keywords": [
        "AI small business",
        "AI automation business",
        "AI tool SMB",
        "ChatGPT business workflow"
      ],
      "sources": [
        "reddit:r/smallbusiness",
        "reddit:r/SaaS",
        "reddit:r/AiForSmallBusiness",
        "hackernews"
      ],
      "extra_signal_keywords": [
        "too expensive",
        "simpler alternative",
        "overkill for my needs"
      ]
    },
    {
      "name": "OpenClaw skills gaps",
      "keywords": [
        "OpenClaw skill",
        "AI agent skill",
        "Codex agent plugin",
        "agent automation"
      ],
      "sources": [
        "reddit:r/openclaw",
        "reddit:r/AI_Agents",
        "hackernews",
        "github:openclawai/OpenClaw"
      ],
      "extra_signal_keywords": [
        "missing skill",
        "wish there was a skill",
        "no plugin for"
      ]
    },
    {
      "name": "Local AI tools",
      "keywords": [
        "local LLM tool",
        "self-hosted AI",
        "local AI workflow",
        "run locally AI"
      ],
      "sources": [
        "reddit:r/LocalLLaMA",
        "reddit:r/selfhosted",
        "hackernews"
      ],
      "extra_signal_keywords": [
        "privacy concerns",
        "offline AI",
        "no cloud dependency",
        "self-hosted alternative"
      ]
    }
  ],
  "scoring_weights": {
    "signal_strength": 0.30,
    "engagement": 0.20,
    "freshness": 0.20,
    "competition": 0.15,
    "recurrence": 0.15
  },
  "max_queries_per_scan": 30,
  "max_results_per_query": 10
}


path: findings/digest-2026-03-26.md
content: ---
date: 2026-03-26
time: 17:36
type: opportunity-scan
niches: [Local AI tools, AI tools for small business]
findings_count: 11
tags: [opportunity-scout, demand-signals, local-ai-tools, ai-tools-for-small-business]
---

# 🔭 Opportunity Scout — 2026-03-26

## Executive Summary

Scanned 11 signals across Local AI tools, AI tools for small business. **8 strong opportunities** scored 7.0+ — these deserve attention. Strongest signal: "Ask HN: What boring business problem are you solving with AI?" (8.8/10).

## 🎯 Top Opportunities

### 1. 🔥 Ask HN: What boring business problem are you solving with AI?

**Score: 8.8/10** ●●●●○
**Source:** [hackernews](https://news.ycombinator.com/item?id=99001)
**Niche:** AI tools for small business

> I cobbled together a Python script that uses GPT-4 to parse my invoices and match them to POs. Ugly workaround but saves me 3 hours/week. Why is there no proper tool for this? Would gladly pay if someone productized it. 203 points, 147 comments.

**Signal Analysis:** Signal: 10 · Engage: 9 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** workaround, why is there no

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 2. 🔥 Alternative to Grammarly that runs locally? Privacy concerns with sending everything I write to the cloud

**Score: 8.8/10** ●●●●○
**Source:** [reddit:r/selfhosted](https://reddit.com/r/selfhosted/comments/bcd890/local_grammarly_alternative)
**Niche:** Local AI tools

> I write sensitive legal documents and I'm not comfortable sending them through Grammarly's servers. Anyone solved this? I'd pay for a good local grammar/style checker. Current workaround: I paste into a local LLM chat but it's clunky. 92 upvotes, 41 comments.

**Signal Analysis:** Signal: 10 · Engage: 8 · Fresh: 10 · No-Comp: 9 · Recur: 6
**Matched signals:** alternative to, anyone solved

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 3. 🔥 I wish there was a simple AI tool that could handle my bookkeeping without needing a CPA

**Score: 8.5/10** ●●●●○
**Source:** [reddit:r/smallbusiness](https://reddit.com/r/smallbusiness/comments/abc123/i_wish_there_was_a_simple_ai_tool)
**Niche:** AI tools for small business

> I've been running a landscaping business for 5 years. Every month I spend 8 hours on bookkeeping. I've tried QuickBooks and it's overkill. Wave is better but still clunky. I wish someone would build a dead simple AI that just categorizes my bank transactions and generates a P&L. I'd pay $20/month...

**Signal Analysis:** Signal: 10 · Engage: 8 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** I wish

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 4. 🔥 Frustrated with Zapier pricing - need alternative for simple automations

**Score: 8.4/10** ●●●●○
**Source:** [reddit:r/SaaS](https://reddit.com/r/SaaS/comments/jkl012/frustrated_zapier_pricing)
**Niche:** AI tools for small business

> I have 5 simple zaps and they want $50/month. All I'm doing is syncing a Google Form to a Notion database and sending a Slack notification. Has anyone solved this without spending a fortune? I'd pay for something simpler. 156 upvotes, 72 comments.

**Signal Analysis:** Signal: 9 · Engage: 9 · Fresh: 10 · No-Comp: 7 · Recur: 6
**Matched signals:** frustrated, anyone solved

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 5. ⚡ Someone should build a simple CRM for solo consultants

**Score: 8.0/10** ●●●●○
**Source:** [reddit:r/smallbusiness](https://reddit.com/r/smallbusiness/comments/ghi789/someone_should_build_simple_crm)
**Niche:** AI tools for small business

> I don't need Salesforce. I don't need HubSpot. I need something that tracks my 30 clients, reminds me to follow up, and lets me log notes. Every CRM out there is designed for sales teams of 50. Someone should build something for the rest of us. Frustrated that this doesn't exist. 89 upvotes, 45 c...

**Signal Analysis:** Signal: 8 · Engage: 8 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** someone should build

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 6. ⚡ I wish there was a local AI coding assistant that actually worked offline

**Score: 7.8/10** ●●●●○
**Source:** [reddit:r/LocalLLaMA](https://reddit.com/r/LocalLLaMA/comments/pqr678/local_coding_assistant_offline)
**Niche:** Local AI tools

> Copilot is great but I work on classified projects where I can't send code to the cloud. I've tried running CodeLlama locally but the experience is nowhere near Copilot. I wish someone would build a polished local coding assistant with good IDE integration. Frustrated that all the good tools requ...

**Signal Analysis:** Signal: 8 · Engage: 8 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** I wish, frustrated

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 7. ⚡ Looking for AI that can write job postings that don't sound robotic

**Score: 7.6/10** ●●●●○
**Source:** [reddit:r/smallbusiness](https://reddit.com/r/smallbusiness/comments/def456/looking_for_ai_job_postings)
**Niche:** AI tools for small business

> Every AI writing tool I've tried produces the same corporate buzzword garbage. I need something that can match my company's casual tone. Has anyone found a good alternative to ChatGPT for this specific use case? 12 upvotes, 8 comments.

**Signal Analysis:** Signal: 8 · Engage: 6 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** looking for

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 8. ⚡ Why is there no good self-hosted alternative to Typeform?

**Score: 7.1/10** ●●●●○
**Source:** [reddit:r/SaaS](https://reddit.com/r/SaaS/comments/mno345/self_hosted_typeform_alternative)
**Niche:** AI tools for small business

> I need beautiful forms that I can embed on my site. Typeform wants $30/month for basic features. Why is there no open-source version that looks as good? The self-hosted options all look like they're from 2005. 34 upvotes, 19 comments.

**Signal Analysis:** Signal: 6 · Engage: 7 · Fresh: 8 · No-Comp: 9 · Recur: 6
**Matched signals:** alternative to

**Why it matters:** High signal strength with low existing competition suggests a genuine unmet need worth exploring.
**Next step:** Deep-dive research — check the thread replies, search for existing solutions, estimate market size.

---

### 9. ⚡ Has anyone found a good local RAG solution that handles PDFs well?

**Score: 7.0/10** ●●●○○
**Source:** [reddit:r/LocalLLaMA](https://reddit.com/r/LocalLLaMA/comments/stu901/local_rag_pdf_solution)
**Niche:** Local AI tools

> I've tried 5 different RAG setups. They all choke on tables and images in PDFs. Looking for something that can actually parse complex documents without cloud APIs. My current workaround is manually extracting text which defeats the purpose. 45 upvotes, 28 comments.

**Signal Analysis:** Signal: 5 · Engage: 8 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** looking for

**Why it matters:** Moderate demand signal — worth monitoring for recurrence across future scans.
**Next step:** Add to watchlist. If this signal persists across 2-3 more scans, investigate further.

---

### 10. ⚡ Need a self-hosted tool for meeting transcription that's actually accurate

**Score: 6.5/10** ●●●○○
**Source:** [reddit:r/selfhosted](https://reddit.com/r/selfhosted/comments/yza567/self_hosted_meeting_transcription)
**Niche:** Local AI tools

> Otter.ai is great but I can't send client meetings to a third party. Whisper is OK for English but terrible for mixed-language meetings. Need a tool that handles multiple languages and speaker diarization locally. 67 upvotes, 31 comments.

**Signal Analysis:** Signal: 3 · Engage: 8 · Fresh: 9 · No-Comp: 9 · Recur: 6
**Matched signals:** need a tool

**Why it matters:** Moderate demand signal — worth monitoring for recurrence across future scans.
**Next step:** Add to watchlist. If this signal persists across 2-3 more scans, investigate further.

---

## 📋 All Findings (Appendix)

*1 additional findings below the top 10 cutoff.*

| # | Score | Title | Source |
|---|-------|-------|--------|
| 11 | 5.2 | [Show r/LocalLLaMA: My local AI email assistant](https://reddit.com/r/LocalLLaMA/comments/vwx234/local_email_assistant) | reddit:r/LocalLLaMA |

---
*Generated by Opportunity Scout on 2026-03-26 at 17:36 ET*

path: references/signal-types.md
content: # Demand Signal Types

A taxonomy of signals that indicate real, actionable market demand. Use this when
scoring findings to distinguish genuine opportunity from noise.

## Table of Contents

1. [Direct Demand Signals](#direct-demand-signals)
2. [Indirect Demand Signals](#indirect-demand-signals)
3. [Noise Signals (Filter Out)](#noise-signals)
4. [Signal Strength Indicators](#signal-strength-indicators)
5. [Examples by Signal Type](#examples)

---

## Direct Demand Signals

These explicitly state an unmet need. Highest confidence.

### 1. Explicit Wish / Feature Request

Someone directly asks for something that doesn't exist or lacks key features.

**Keyword patterns**: "I wish", "if only", "someone should build", "is there a tool
that", "looking for a way to", "need something that", "does anyone know of"

**Example (Reddit r/smallbusiness)**:
> "I wish there was an invoicing tool that just let me send a PDF and get paid.
> Every tool I try wants me to sign up for a whole ecosystem. I just want simple invoicing."

**Example (HN)**:
> "Is there a self-hosted alternative to Notion that doesn't feel like it was
> designed by committee? I want something fast and local."

**Signal strength**: HIGH — direct expression of unmet need.

### 2. Frustration / Pain Expression

Someone describes a painful experience with existing solutions.

**Keyword patterns**: "frustrated", "annoyed", "hate that", "terrible experience",
"why does every", "so tired of", "fed up with", "drives me crazy"

**Example (Reddit r/SaaS)**:
> "I'm so frustrated with Zapier pricing. I have 5 simple automations and they
> want $50/month. There has to be a better way for small operations."

**Example (Reddit r/selfhosted)**:
> "Why does every self-hosted dashboard assume you're running Kubernetes?
> I have 3 Docker containers on a Pi. I don't need Grafana."

**Signal strength**: HIGH — pain + specific problem = clear opportunity.

### 3. "Shut Up and Take My Money"

Someone explicitly states willingness to pay for a solution.

**Keyword patterns**: "I'd pay for", "shut up and take my money", "worth paying for",
"I'd switch to", "instant buy if", "take my money"

**Example (HN)**:
> "If someone built a Gmail client that was actually fast and didn't spy on me,
> I'd pay $10/month tomorrow. Fastmail is close but the UI is 2010."

**Signal strength**: VERY HIGH — validated willingness to pay. Rare but golden.

### 4. Solution Seeking

Someone actively searching for a solution, hasn't found one.

**Keyword patterns**: "has anyone solved", "how do you handle", "what do you use for",
"best way to", "anyone found a good", "recommendations for"

**Example (Reddit r/LocalLLaMA)**:
> "Has anyone found a good way to do RAG over a large codebase locally?
> Everything I've tried either chokes on context or needs cloud APIs."

**Signal strength**: MEDIUM-HIGH — demand is clear, but might have existing solutions
the poster hasn't found. Check replies for competition signals.

---

## Indirect Demand Signals

These don't directly ask for something but reveal gaps through behavior.

### 5. Workaround Description

Someone describes a hacky, manual, or fragile solution they built. This means:
the need exists, no good solution exists, they're invested enough to build something ugly.

**Keyword patterns**: "workaround", "hack", "duct tape", "jerry-rigged", "I wrote a
script that", "my current workflow is", "cobbled together", "ugly but works"

**Example (Reddit r/selfhosted)**:
> "I cobbled together a bash script that scrapes my ISP's usage page, logs it to
> a CSV, and sends me a Telegram alert when I hit 80%. Ugly but it works."

**Signal strength**: HIGH — someone invested real effort = strong unmet need.
The uglier the workaround, the bigger the opportunity.

### 6. Comparison Shopping / "Alternative To"

Someone comparing existing options and finding all lacking.

**Keyword patterns**: "alternative to", "compared to", "vs", "switching from",
"looking to replace", "moved away from", "outgrown"

**Example (Reddit r/SaaS)**:
> "I've tried Airtable, Notion, and Coda for project management and they all
> feel like Swiss Army knives when I need a screwdriver. Is there anything
> that just does task tracking without the bloat?"

**Signal strength**: MEDIUM-HIGH — dissatisfaction with existing options = opportunity
for a focused competitor.

### 7. Scaling Pain

Someone hit limits of their current solution as they grew.

**Keyword patterns**: "outgrew", "doesn't scale", "worked fine until", "now that we
have more", "hit the limits", "breaking at scale"

**Example (Reddit r/smallbusiness)**:
> "Google Sheets worked fine for inventory when we had 50 SKUs. Now we have 500
> and it's a nightmare. Everything is slow and formulas break constantly."

**Signal strength**: MEDIUM — indicates a market segment (growing businesses) that
existing tools under-serve.

### 8. Migration / Escape

Someone trying to leave a platform, blocked by lock-in or lack of alternatives.

**Keyword patterns**: "migrate from", "escape", "leave", "export my data",
"locked in", "held hostage", "can't switch because"

**Example (HN)**:
> "I want to leave Heroku but every alternative either costs 3x more or requires
> a DevOps hire. There's a massive gap for 'Heroku but cheaper and not dead.'"

**Signal strength**: MEDIUM-HIGH — lock-in frustration often comes with willingness
to pay for escape routes.

---

## Noise Signals

Filter these OUT — they look like demand but aren't actionable.

### Vague Complaints

> "Everything sucks these days" / "Software quality is declining"

No specific problem, no specific need. Skip.

### Already Solved (Check Replies)

If the top replies provide 3+ solid solutions with upvotes, the need is likely met.
Lower the opportunity score significantly.

### One Person's Edge Case

> "I need a tool that converts Esperanto PDFs to Braille-compatible HTML"

Real need, market of one. Not an opportunity unless you see it recurring.

### Hype / Trend Chasing

> "Someone should build an AI agent that does everything with blockchain"

Buzzword soup without specific pain. Usually noise.

### Self-Promotion Disguised as Demand

> "Has anyone tried [specific product with affiliate link]?"

Not a demand signal — it's marketing.

---

## Signal Strength Indicators

Beyond the signal type, these factors amplify or dampen strength:

| Factor | Amplifies | Dampens |
|--------|-----------|---------|
| **Engagement** | 50+ upvotes, 20+ comments | 0-2 upvotes, no discussion |
| **Specificity** | Named tools, dollar amounts, metrics | Vague desires |
| **Recurrence** | Same signal across 3+ posts/weeks | One-off mention |
| **Recency** | Posted within last 30 days | 6+ months old |
| **Author profile** | Active community member, domain expert | New/throwaway account |
| **Reply quality** | "I have this problem too" replies | "Just use X" with solutions |
| **Community fit** | Posted in relevant, active subreddit | Cross-posted spam |

## Quick Scoring Guide

- **9-10**: "Shut up and take my money" + high engagement + recurring + no solution in replies
- **7-8**: Clear demand expression + decent engagement + partial or no solutions available
- **5-6**: Indirect signal (workaround, comparison) + some engagement + solutions exist but are imperfect
- **3-4**: Vague demand + low engagement OR demand is clear but well-served by existing tools
- **1-2**: Noise, one-person edge case, or fully solved problem


path: references/source-guide.md
content: # Source Configuration Guide

How to configure sources effectively for opportunity scanning.

## Supported Source Types

### Reddit

Format: `reddit:r/<subreddit>`

Reddit is the richest source for demand signals. People vent, ask for help, and
describe workarounds in detail.

**Best subreddits by niche:**

- **SaaS / software**: r/SaaS, r/startups, r/EntrepreneurRideAlong, r/microsaas
- **Small business**: r/smallbusiness, r/Entrepreneur, r/ecommerce
- **AI / automation**: r/AI_Agents, r/LocalLLaMA, r/AiForSmallBusiness, r/ChatGPT
- **Self-hosted / developer**: r/selfhosted, r/homelab, r/webdev, r/devops
- **Specific verticals**: r/realestateinvesting, r/freelance, r/accounting

**Tips:**
- Niche subreddits (under 100k members) often have higher signal density than large ones.
- Sort by "new" to catch emerging pain before it gets solved.
- r/SaaS and r/smallbusiness are gold mines for B2B product ideas.

### Hacker News

Format: `hackernews`

HN skews technical and founder-oriented. Good for developer tools, infrastructure,
and technical product gaps.

**HN-specific signals:**
- "Show HN" comment threads often reveal what's missing from presented solutions.
- "Ask HN" threads starting with "What tool do you wish existed?" are pure signal.
- Comments on product launches that say "I want this but for X" = adjacent opportunities.

### GitHub Issues

Format: `github:<owner>/<repo>`

Open issues and feature requests on popular repos are direct demand signals from
users who tried the tool and found it lacking.

**Tips:**
- Sort by most thumbs-up reactions — this surfaces the most-wanted features.
- Issues with "help wanted" or "good first issue" labels but no PRs = opportunity.
- Feature request issues that keep getting reopened = persistent unmet demand.

## Choosing Sources for Your Niche

**Start with 3-5 sources per niche.** More sources = more noise, not necessarily
more signal. Quality of subreddit selection matters more than quantity.

**Strategy:**
1. Pick 1-2 subreddits where your target users actually hang out
2. Add HN if your niche is technical
3. Add 1 GitHub repo if there's a dominant open-source tool in the space
4. Monitor results for 2-3 scans, then adjust: drop noisy sources, add discovered ones

## Keyword Configuration

### Signal Keywords (Required)

These identify demand expressions. The defaults cover most cases:
`wish, need, looking for, alternative to, frustrated, anyone solved, workaround,
hate that, shut up and take my money, I'd pay for, why is there no, someone should build`

### Niche Keywords (Required)

These scope the search to your domain. Be specific:
- Good: "invoicing small business", "self-hosted dashboard"
- Bad: "software", "app" (too broad, drowns in noise)

### Combining Keywords

The scanner combines niche + signal keywords into queries like:
`site:reddit.com/r/SaaS "I wish" invoicing small business`

This targets demand expressions within your specific domain and community.

## Scan Frequency

- **Daily**: High-velocity niches (AI, crypto, trending tech). Catches signals fast
  but generates more to review.
- **Weekly**: Stable niches (B2B tools, self-hosted). Less noise, still catches
  persistent signals. Recommended starting point.

## Avoiding Noise

1. **Be specific in niche keywords** — broad terms flood results with irrelevant posts
2. **Curate subreddits** — one well-chosen subreddit beats five vaguely related ones
3. **Review and adjust** — after each scan, note which sources produced signal vs noise
4. **Use history tracking** — recurring signals across scans are more reliable than one-offs


path: scripts/configure.py
content: #!/usr/bin/env python3
"""Configure opportunity-scout niches, keywords, sources, and schedule.

Usage:
    configure.py --init                          Copy example config to config.json
    configure.py --add-niche "Name" --keywords "k1,k2,k3"
    configure.py --add-source reddit:r/SaaS,hackernews [--niche "Name"]
    configure.py --remove-niche "Name"
    configure.py --set-schedule daily|weekly
    configure.py --set-weight signal_strength=0.3,engagement=0.2
    configure.py --show                          Print current config
"""

import argparse
import json
import os
import shutil
import sys

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CONFIG_PATH = os.path.join(SKILL_DIR, "config.json")
EXAMPLE_PATH = os.path.join(SKILL_DIR, "assets", "config.example.json")


def load_config():
    """Load config.json or exit with helpful message."""
    if not os.path.exists(CONFIG_PATH):
        print(f"Error: {CONFIG_PATH} not found.", file=sys.stderr)
        print("Run: configure.py --init  (copies example config)", file=sys.stderr)
        sys.exit(1)
    with open(CONFIG_PATH, "r") as f:
        return json.load(f)


def save_config(config):
    """Write config back to disk."""
    with open(CONFIG_PATH, "w") as f:
        json.dump(config, f, indent=2)
    print(f"Config saved to {CONFIG_PATH}")


def init_config():
    """Copy example config to config.json."""
    if os.path.exists(CONFIG_PATH):
        print(f"Config already exists at {CONFIG_PATH}")
        print("Delete it first or edit directly.")
        sys.exit(1)
    if not os.path.exists(EXAMPLE_PATH):
        print(f"Error: Example config not found at {EXAMPLE_PATH}", file=sys.stderr)
        sys.exit(1)
    shutil.copy2(EXAMPLE_PATH, CONFIG_PATH)
    print(f"Initialized config from example: {CONFIG_PATH}")


def add_niche(config, name, keywords, sources=None, extra_signal_keywords=None):
    """Add a new niche or update existing one."""
    # Check if niche exists
    for niche in config.get("niches", []):
        if niche["name"].lower() == name.lower():
            print(f"Updating existing niche: {name}")
            if keywords:
                niche["keywords"] = keywords
            if sources:
                existing = set(niche.get("sources", []))
                existing.update(sources)
                niche["sources"] = sorted(existing)
            if extra_signal_keywords:
                niche["extra_signal_keywords"] = extra_signal_keywords
            return config

    # Create new niche
    niche = {
        "name": name,
        "keywords": keywords if keywords else [],
        "sources": sources if sources else [],
    }
    if extra_signal_keywords:
        niche["extra_signal_keywords"] = extra_signal_keywords
    config.setdefault("niches", []).append(niche)
    print(f"Added niche: {name}")
    return config


def remove_niche(config, name):
    """Remove a niche by name."""
    niches = config.get("niches", [])
    original_len = len(niches)
    config["niches"] = [n for n in niches if n["name"].lower() != name.lower()]
    if len(config["niches"]) < original_len:
        print(f"Removed niche: {name}")
    else:
        print(f"Niche not found: {name}", file=sys.stderr)
        sys.exit(1)
    return config


def add_sources(config, sources, niche_name=None):
    """Add sources to a specific niche or all niches."""
    niches = config.get("niches", [])
    if not niches:
        print("Error: No niches configured. Add a niche first.", file=sys.stderr)
        sys.exit(1)

    targets = niches
    if niche_name:
        targets = [n for n in niches if n["name"].lower() == niche_name.lower()]
        if not targets:
            print(f"Error: Niche '{niche_name}' not found.", file=sys.stderr)
            sys.exit(1)

    for niche in targets:
        existing = set(niche.get("sources", []))
        existing.update(sources)
        niche["sources"] = sorted(existing)
        print(f"Updated sources for '{niche['name']}': {niche['sources']}")

    return config


def set_schedule(config, schedule):
    """Set scan schedule."""
    if schedule not in ("daily", "weekly"):
        print("Error: Schedule must be 'daily' or 'weekly'.", file=sys.stderr)
        sys.exit(1)
    config["schedule"] = schedule
    print(f"Schedule set to: {schedule}")
    return config


def set_weights(config, weight_str):
    """Set scoring weights from comma-separated key=value pairs."""
    valid_keys = {"signal_strength", "engagement", "freshness", "competition", "recurrence"}
    weights = config.get("scoring_weights", {})

    for pair in weight_str.split(","):
        pair = pair.strip()
        if "=" not in pair:
            print(f"Error: Invalid weight format '{pair}'. Use key=value.", file=sys.stderr)
            sys.exit(1)
        key, val = pair.split("=", 1)
        key = key.strip()
        if key not in valid_keys:
            print(f"Error: Unknown weight '{key}'. Valid: {valid_keys}", file=sys.stderr)
            sys.exit(1)
        try:
            weights[key] = float(val)
        except ValueError:
            print(f"Error: Weight value must be a number: '{val}'", file=sys.stderr)
            sys.exit(1)

    total = sum(weights.values())
    if abs(total - 1.0) > 0.01:
        print(f"Warning: Weights sum to {total:.2f}, not 1.0. Consider rebalancing.")

    config["scoring_weights"] = weights
    print(f"Scoring weights updated: {weights}")
    return config


def show_config(config):
    """Pretty-print current configuration."""
    print(json.dumps(config, indent=2))


def main():
    parser = argparse.ArgumentParser(description="Configure opportunity-scout")
    parser.add_argument("--init", action="store_true", help="Initialize config from example")
    parser.add_argument("--add-niche", metavar="NAME", help="Add or update a niche")
    parser.add_argument("--keywords", help="Comma-separated niche keywords")
    parser.add_argument("--extra-signal-keywords", help="Comma-separated extra signal keywords")
    parser.add_argument("--add-source", help="Comma-separated sources to add")
    parser.add_argument("--niche", help="Target niche for --add-source")
    parser.add_argument("--remove-niche", metavar="NAME", help="Remove a niche")
    parser.add_argument("--set-schedule", choices=["daily", "weekly"], help="Set scan schedule")
    parser.add_argument("--set-weight", metavar="WEIGHTS", help="Set scoring weights (key=val,...)")
    parser.add_argument("--show", action="store_true", help="Show current config")

    args = parser.parse_args()

    if args.init:
        init_config()
        return

    if not any([args.add_niche, args.add_source, args.remove_niche,
                args.set_schedule, args.set_weight, args.show]):
        parser.print_help()
        return

    config = load_config()

    if args.show:
        show_config(config)
        return

    if args.add_niche:
        keywords = [k.strip() for k in args.keywords.split(",")] if args.keywords else []
        extra = [k.strip() for k in args.extra_signal_keywords.split(",")] if args.extra_signal_keywords else None
        sources = [s.strip() for s in args.add_source.split(",")] if args.add_source else None
        config = add_niche(config, args.add_niche, keywords, sources, extra)
    elif args.add_source:
        sources = [s.strip() for s in args.add_source.split(",")]
        config = add_sources(config, sources, args.niche)

    if args.remove_niche:
        config = remove_niche(config, args.remove_niche)

    if args.set_schedule:
        config = set_schedule(config, args.set_schedule)

    if args.set_weight:
        config = set_weights(config, args.set_weight)

    save_config(config)


if __name__ == "__main__":
    main()


path: scripts/digest.py
content: #!/usr/bin/env python3
"""Generate a prioritized markdown digest from scored opportunity findings.

Usage:
    digest.py <scored.json>                      Print digest to stdout
    digest.py <scored.json> --output report.md   Save to file
    digest.py <scored.json> --max-results 15     Limit top opportunities shown
    digest.py - < scored.json                    Read from stdin
"""

import argparse
import json
import os
import sys
from datetime import datetime

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HISTORY_PATH = os.path.join(SKILL_DIR, "history.json")


def load_history():
    """Load history for trend data."""
    if not os.path.exists(HISTORY_PATH):
        return None
    try:
        with open(HISTORY_PATH, "r") as f:
            return json.load(f)
    except (json.JSONDecodeError, IOError):
        return None


def classify_signal(sig):
    """Classify a signal as persistent, emerging, or fading."""
    scan_count = len(sig.get("scan_dates", []))
    scores = sig.get("scores_over_time", [])
    if scan_count >= 3:
        if len(scores) >= 3:
            recent_avg = sum(s["score"] for s in scores[-2:]) / 2
            older_avg = sum(s["score"] for s in scores[:-2]) / max(len(scores) - 2, 1)
            if recent_avg < older_avg * 0.7:
                return "fading"
        return "persistent"
    return "emerging"


def score_bar(score, max_score=10.0):
    """Generate a visual bar for a score."""
    filled = int(round(score / max_score * 5))
    return "●" * filled + "○" * (5 - filled)


def format_score_breakdown(scores):
    """Format score dimensions as a compact line."""
    parts = []
    labels = {
        "signal_strength": "Signal",
        "engagement": "Engage",
        "freshness": "Fresh",
        "competition": "No-Comp",
        "recurrence": "Recur",
    }
    for key, label in labels.items():
        val = scores.get(key, 0)
        parts.append(f"{label}: {val:.0f}")
    return " · ".join(parts)


def generate_executive_summary(scored_findings):
    """Generate 2-3 sentence executive summary."""
    if not scored_findings:
        return "No signals found in this scan cycle."

    top = scored_findings[:3]
    niches = list(set(f.get("niche", "General") for f in scored_findings if f.get("niche")))
    niche_str = ", ".join(niches[:3]) if niches else "configured niches"

    strong_count = sum(1 for f in scored_findings if f.get("composite_score", 0) >= 7.0)
    total = len(scored_findings)

    lines = []
    lines.append(f"Scanned {total} signals across {niche_str}.")

    if strong_count > 0:
        lines.append(f"**{strong_count} strong opportunities** scored 7.0+ — these deserve attention.")
    else:
        lines.append("No standout signals this cycle — the findings below are moderate strength.")

    if top:
        top_title = top[0].get("title", "Unknown")
        top_score = top[0].get("composite_score", 0)
        lines.append(f"Strongest signal: \"{top_title}\" ({top_score:.1f}/10).")

    return " ".join(lines)


def generate_digest(scored_findings, max_results=20):
    """Generate the full markdown digest."""
    now = datetime.now()
    date_str = now.strftime("%Y-%m-%d")
    time_str = now.strftime("%H:%M")

    niches = list(set(f.get("niche", "") for f in scored_findings if f.get("niche")))
    sources = list(set(f.get("source", "") for f in scored_findings if f.get("source")))

    lines = []

    # YAML frontmatter
    lines.append("---")
    lines.append(f"date: {date_str}")
    lines.append(f"time: {time_str}")
    lines.append("type: opportunity-scan")
    niche_yaml = ", ".join(niches) if niches else "general"
    lines.append(f"niches: [{niche_yaml}]")
    lines.append(f"findings_count: {len(scored_findings)}")

    # Auto-generate tags
    tags = ["opportunity-scout", "demand-signals"]
    for n in niches:
        tag = n.lower().replace(" ", "-")[:30]
        tags.append(tag)
    tags_yaml = ", ".join(tags)
    lines.append(f"tags: [{tags_yaml}]")
    lines.append("---")
    lines.append("")

    # Title
    lines.append(f"# 🔭 Opportunity Scout — {date_str}")
    lines.append("")

    # Executive Summary
    lines.append("## Executive Summary")
    lines.append("")
    lines.append(generate_executive_summary(scored_findings))
    lines.append("")

    # Top Opportunities
    top = scored_findings[:max_results]
    lines.append(f"## 🎯 Top Opportunities")
    lines.append("")

    for i, finding in enumerate(top, 1):
        title = finding.get("title", "Untitled")
        url = finding.get("url", "")
        snippet = finding.get("snippet", "")
        composite = finding.get("composite_score", 0)
        scores = finding.get("scores", {})
        source = finding.get("source", "unknown")
        matched = finding.get("matched_keywords", [])
        niche = finding.get("niche", "")

        # Score tier
        if composite >= 8.0:
            tier = "🔥"
        elif composite >= 6.0:
            tier = "⚡"
        elif composite >= 4.0:
            tier = "💡"
        else:
            tier = "📌"

        lines.append(f"### {i}. {tier} {title}")
        lines.append("")
        lines.append(f"**Score: {composite:.1f}/10** {score_bar(composite)}")
        lines.append(f"**Source:** [{source}]({url})")
        if niche:
            lines.append(f"**Niche:** {niche}")
        lines.append("")

        if snippet:
            # Truncate long snippets
            if len(snippet) > 300:
                snippet = snippet[:297] + "..."
            lines.append(f"> {snippet}")
            lines.append("")

        lines.append(f"**Signal Analysis:** {format_score_breakdown(scores)}")
        if matched:
            lines.append(f"**Matched signals:** {', '.join(matched[:5])}")
        lines.append("")

        # Why it matters / suggested next step
        if composite >= 7.0:
            lines.append("**Why it matters:** High signal strength with low existing "
                         "competition suggests a genuine unmet need worth exploring.")
            lines.append("**Next step:** Deep-dive research — check the thread replies, "
                         "search for existing solutions, estimate market size.")
        elif composite >= 5.0:
            lines.append("**Why it matters:** Moderate demand signal — worth monitoring "
                         "for recurrence across future scans.")
            lines.append("**Next step:** Add to watchlist. If this signal persists "
                         "across 2-3 more scans, investigate further.")
        else:
            lines.append("**Why it matters:** Weak or partially-served signal. "
                         "Keep on radar but don't invest research time yet.")

        lines.append("")
        lines.append("---")
        lines.append("")

    # Recurring Signals (from history)
    history = load_history()
    if history and history.get("signals"):
        persistent = []
        fading = []
        for key, sig in history["signals"].items():
            classification = classify_signal(sig)
            if classification == "persistent":
                persistent.append(sig)
            elif classification == "fading":
                fading.append(sig)

        if persistent:
            persistent.sort(key=lambda x: x.get("peak_score", 0), reverse=True)
            lines.append("## 🔁 Recurring Signals")
            lines.append("")
            lines.append("*These signals keep appearing across scans — persistent "
                         "demand is a strong indicator of real opportunity.*")
            lines.append("")
            for sig in persistent[:10]:
                title = sig["titles"][0] if sig.get("titles") else "Unknown"
                peak = sig.get("peak_score", 0)
                count = len(sig.get("scan_dates", []))
                first = sig.get("first_seen", "?")
                lines.append(f"- **{title}** — score {peak:.1f}, seen {count}x since {first}")
            lines.append("")

        if fading:
            fading.sort(key=lambda x: x.get("peak_score", 0), reverse=True)
            lines.append("## 📉 Fading Signals")
            lines.append("")
            lines.append("*Previously strong signals that are declining — the market "
                         "may be filling these gaps.*")
            lines.append("")
            for sig in fading[:10]:
                title = sig["titles"][0] if sig.get("titles") else "Unknown"
                peak = sig.get("peak_score", 0)
                last = sig.get("last_seen", "?")
                lines.append(f"- **{title}** — peak score {peak:.1f}, last seen {last}")
            lines.append("")

    # Raw Findings appendix
    if len(scored_findings) > max_results:
        remaining = scored_findings[max_results:]
        lines.append("## 📋 All Findings (Appendix)")
        lines.append("")
        lines.append(f"*{len(remaining)} additional findings below the top {max_results} cutoff.*")
        lines.append("")
        lines.append("| # | Score | Title | Source |")
        lines.append("|---|-------|-------|--------|")
        for i, f in enumerate(remaining, max_results + 1):
            title = f.get("title", "Untitled")[:60]
            score = f.get("composite_score", 0)
            source = f.get("source", "?")
            url = f.get("url", "")
            lines.append(f"| {i} | {score:.1f} | [{title}]({url}) | {source} |")
        lines.append("")

    # Footer
    lines.append("---")
    lines.append(f"*Generated by Opportunity Scout on {date_str} at {time_str} ET*")

    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(description="Generate opportunity digest")
    parser.add_argument("scored", help="Path to scored findings JSON (or - for stdin)")
    parser.add_argument("--output", "-o", help="Output file path")
    parser.add_argument("--max-results", type=int, default=20,
                        help="Max top opportunities to show (default: 20)")

    args = parser.parse_args()

    # Load scored findings
    if args.scored == "-":
        scored = json.load(sys.stdin)
    else:
        with open(args.scored, "r") as f:
            scored = json.load(f)

    digest = generate_digest(scored, args.max_results)

    if args.output:
        os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True)
        with open(args.output, "w") as f:
            f.write(digest)
        print(f"Digest saved to {args.output}")
    else:
        print(digest)


if __name__ == "__main__":
    main()


path: scripts/history.py
content: #!/usr/bin/env python3
"""Track opportunity signals over time and detect trend patterns.

Usage:
    history.py --update <scored.json>     Add scored findings to history
    history.py --trends                   Show persistent, emerging, and fading signals
    history.py --trends --format json     Machine-readable trend output
    history.py --timeline <topic>         Show timeline for a specific signal topic
    history.py --stats                    Show history statistics
"""

import argparse
import json
import os
import re
import sys
from datetime import datetime

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HISTORY_PATH = os.path.join(SKILL_DIR, "history.json")


def load_history():
    """Load history.json or return empty structure."""
    if not os.path.exists(HISTORY_PATH):
        return {
            "version": 1,
            "scan_log": [],
            "signals": {},
        }
    try:
        with open(HISTORY_PATH, "r") as f:
            return json.load(f)
    except (json.JSONDecodeError, IOError):
        return {
            "version": 1,
            "scan_log": [],
            "signals": {},
        }


def save_history(history):
    """Write history back to disk."""
    with open(HISTORY_PATH, "w") as f:
        json.dump(history, f, indent=2)


def normalize_title(title):
    """Create a normalized key from a title for fuzzy matching across scans."""
    # Lowercase, remove punctuation, strip common words
    title = title.lower().strip()
    title = re.sub(r'[^\w\s]', '', title)
    stop_words = {"the", "a", "an", "is", "are", "was", "were", "for", "to", "in",
                  "on", "of", "and", "or", "but", "with", "how", "what", "why",
                  "has", "anyone", "does", "do", "i", "my", "there", "this", "that",
                  "it", "be", "have", "can", "just", "so", "not", "any"}
    words = [w for w in title.split() if w not in stop_words and len(w) > 2]
    # Take up to 8 meaningful words as the key
    return " ".join(sorted(words[:8]))


def update_history(history, scored_findings):
    """Add scored findings to history, updating signal tracking."""
    scan_date = datetime.now().strftime("%Y-%m-%d")

    # Log the scan
    scan_entry = {
        "date": scan_date,
        "findings_count": len(scored_findings),
        "niches": list(set(f.get("niche", "") for f in scored_findings if f.get("niche"))),
    }
    history["scan_log"].append(scan_entry)

    # Process each finding
    for finding in scored_findings:
        title = finding.get("title", "")
        if not title:
            continue

        key = normalize_title(title)
        if not key:
            continue

        if key not in history["signals"]:
            history["signals"][key] = {
                "first_seen": scan_date,
                "last_seen": scan_date,
                "scan_dates": [scan_date],
                "titles": [title],
                "urls": [finding.get("url", "")],
                "niches": [finding.get("niche", "")],
                "peak_score": finding.get("composite_score", 0),
                "scores_over_time": [{
                    "date": scan_date,
                    "score": finding.get("composite_score", 0),
                }],
            }
        else:
            sig = history["signals"][key]
            sig["last_seen"] = scan_date

            # Only add scan_date if not already present (same-day re-scan)
            if scan_date not in sig["scan_dates"]:
                sig["scan_dates"].append(scan_date)

            # Track title variants
            if title not in sig["titles"]:
                sig["titles"].append(title)

            # Track URLs
            url = finding.get("url", "")
            if url and url not in sig["urls"]:
                sig["urls"].append(url)

            # Track niches
            niche = finding.get("niche", "")
            if niche and niche not in sig["niches"]:
                sig["niches"].append(niche)

            # Update peak score
            score = finding.get("composite_score", 0)
            sig["peak_score"] = max(sig["peak_score"], score)
            sig["scores_over_time"].append({
                "date": scan_date,
                "score": score,
            })

    return history


def classify_signal(sig):
    """Classify a signal as persistent, emerging, fading, or one-off."""
    scan_count = len(sig.get("scan_dates", []))
    scores = sig.get("scores_over_time", [])

    if scan_count >= 3:
        # Check if scores are declining
        if len(scores) >= 3:
            recent_avg = sum(s["score"] for s in scores[-2:]) / 2
            older_avg = sum(s["score"] for s in scores[:-2]) / max(len(scores) - 2, 1)
            if recent_avg < older_avg * 0.7:
                return "fading"
        return "persistent"
    elif scan_count == 1:
        return "emerging"
    elif scan_count == 2:
        # Check recency — if both scans are recent, emerging; if spread out, watch
        return "emerging"

    return "one-off"


def get_trends(history):
    """Analyze all signals and classify into trend categories."""
    trends = {
        "persistent": [],   # 3+ scans, stable or growing
        "emerging": [],      # 1-2 scans, new
        "fading": [],        # Was persistent, now declining
    }

    for key, sig in history.get("signals", {}).items():
        classification = classify_signal(sig)
        entry = {
            "topic": key,
            "titles": sig.get("titles", [])[:3],
            "first_seen": sig.get("first_seen", ""),
            "last_seen": sig.get("last_seen", ""),
            "scan_count": len(sig.get("scan_dates", [])),
            "peak_score": sig.get("peak_score", 0),
            "niches": sig.get("niches", []),
            "urls": sig.get("urls", [])[:3],
        }

        if classification in trends:
            trends[classification].append(entry)

    # Sort each category by peak score descending
    for cat in trends:
        trends[cat].sort(key=lambda x: x["peak_score"], reverse=True)

    return trends


def show_trends(history, output_format="text"):
    """Display trend analysis."""
    trends = get_trends(history)

    if output_format == "json":
        print(json.dumps(trends, indent=2))
        return

    total_signals = len(history.get("signals", {}))
    total_scans = len(history.get("scan_log", []))
    print(f"\n📊 Signal Trends ({total_signals} signals tracked across {total_scans} scans)\n")

    if trends["persistent"]:
        print("🔥 PERSISTENT SIGNALS (appearing 3+ scans — strong demand)")
        print("-" * 60)
        for s in trends["persistent"][:10]:
            title = s["titles"][0] if s["titles"] else s["topic"]
            print(f"  [{s['peak_score']:.1f}] {title}")
            print(f"       Seen {s['scan_count']}x | First: {s['first_seen']} | "
                  f"Niches: {', '.join(s['niches'][:3])}")
        print()

    if trends["emerging"]:
        print("🌱 EMERGING SIGNALS (new — watch these)")
        print("-" * 60)
        for s in trends["emerging"][:10]:
            title = s["titles"][0] if s["titles"] else s["topic"]
            print(f"  [{s['peak_score']:.1f}] {title}")
            print(f"       First seen: {s['first_seen']} | "
                  f"Niches: {', '.join(s['niches'][:3])}")
        print()

    if trends["fading"]:
        print("📉 FADING SIGNALS (declining — market may be filling)")
        print("-" * 60)
        for s in trends["fading"][:10]:
            title = s["titles"][0] if s["titles"] else s["topic"]
            print(f"  [{s['peak_score']:.1f}] {title}")
            print(f"       Seen {s['scan_count']}x | Last: {s['last_seen']}")
        print()

    if not any(trends.values()):
        print("No trend data yet. Run a few scans to build history.")


def show_timeline(history, topic):
    """Show timeline for a specific signal topic."""
    topic_lower = topic.lower()
    for key, sig in history.get("signals", {}).items():
        # Check if topic matches key or any title
        if (topic_lower in key or
                any(topic_lower in t.lower() for t in sig.get("titles", []))):
            print(f"\n📈 Timeline: {sig['titles'][0] if sig['titles'] else key}")
            print(f"   First seen: {sig['first_seen']}")
            print(f"   Last seen: {sig['last_seen']}")
            print(f"   Total appearances: {len(sig['scan_dates'])}")
            print(f"   Peak score: {sig['peak_score']:.1f}")
            print(f"   Status: {classify_signal(sig)}")
            print(f"\n   Score over time:")
            for entry in sig.get("scores_over_time", []):
                bar = "█" * int(entry["score"])
                print(f"     {entry['date']}: {entry['score']:.1f} {bar}")
            print(f"\n   URLs:")
            for url in sig.get("urls", [])[:5]:
                print(f"     {url}")
            return

    print(f"No signals found matching '{topic}'")


def show_stats(history):
    """Show history statistics."""
    signals = history.get("signals", {})
    scans = history.get("scan_log", [])

    print(f"\n📊 History Stats")
    print(f"   Total scans: {len(scans)}")
    print(f"   Total unique signals: {len(signals)}")

    if scans:
        print(f"   First scan: {scans[0].get('date', 'unknown')}")
        print(f"   Last scan: {scans[-1].get('date', 'unknown')}")

    if signals:
        classifications = {"persistent": 0, "emerging": 0, "fading": 0, "one-off": 0}
        for sig in signals.values():
            c = classify_signal(sig)
            classifications[c] = classifications.get(c, 0) + 1
        print(f"\n   Signal breakdown:")
        for c, count in sorted(classifications.items(), key=lambda x: -x[1]):
            print(f"     {c}: {count}")


def main():
    parser = argparse.ArgumentParser(description="Track opportunity signals over time")
    parser.add_argument("--update", metavar="FILE", help="Add scored findings to history")
    parser.add_argument("--trends", action="store_true", help="Show trend analysis")
    parser.add_argument("--timeline", metavar="TOPIC", help="Show timeline for a topic")
    parser.add_argument("--stats", action="store_true", help="Show history statistics")
    parser.add_argument("--format", choices=["text", "json"], default="text",
                        help="Output format for --trends")

    args = parser.parse_args()

    if not any([args.update, args.trends, args.timeline, args.stats]):
        parser.print_help()
        return

    history = load_history()

    if args.update:
        if args.update == "-":
            scored = json.load(sys.stdin)
        else:
            with open(args.update, "r") as f:
                scored = json.load(f)
        history = update_history(history, scored)
        save_history(history)
        print(f"History updated: {len(scored)} findings logged.")
        print(f"Total unique signals tracked: {len(history.get('signals', {}))}")

    if args.trends:
        show_trends(history, args.format)

    if args.timeline:
        show_timeline(history, args.timeline)

    if args.stats:
        show_stats(history)


if __name__ == "__main__":
    main()


path: scripts/scan_sources.py
content: #!/usr/bin/env python3
"""Generate search queries and process raw search results into findings.

Usage:
    scan_sources.py --generate-queries              Output JSON queries for web_search
    scan_sources.py --generate-queries --niche "X"  Queries for one niche only
    scan_sources.py --quick "topic description"     Quick queries without full config
    scan_sources.py --ingest <results.json>          Parse search results into findings
    scan_sources.py --ingest - < results.json        Read from stdin
"""

import argparse
import json
import os
import sys
from datetime import datetime

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CONFIG_PATH = os.path.join(SKILL_DIR, "config.json")

# Default signal keywords if not in config
DEFAULT_SIGNAL_KEYWORDS = [
    "I wish", "someone should build", "looking for", "alternative to",
    "frustrated", "anyone solved", "workaround", "why is there no",
    "I'd pay for", "need a tool", "has anyone found",
]


def load_config():
    """Load config.json."""
    if not os.path.exists(CONFIG_PATH):
        print(f"Error: {CONFIG_PATH} not found. Run configure.py --init", file=sys.stderr)
        sys.exit(1)
    with open(CONFIG_PATH, "r") as f:
        return json.load(f)


def source_to_site_filter(source):
    """Convert source format to search query site filter."""
    if source.startswith("reddit:r/"):
        subreddit = source.replace("reddit:r/", "")
        return f"site:reddit.com/r/{subreddit}"
    elif source == "hackernews":
        return "site:news.ycombinator.com"
    elif source.startswith("github:"):
        repo = source.replace("github:", "")
        return f"site:github.com/{repo}"
    return ""


def generate_queries_for_niche(niche, signal_keywords, max_queries=10):
    """Generate optimized search queries for a single niche.

    Strategy: batch signal keywords into groups and combine with niche keywords
    and source filters. Fewer, broader queries > many narrow ones.
    """
    queries = []
    niche_kws = niche.get("keywords", [])
    sources = niche.get("sources", [])
    extra_signals = niche.get("extra_signal_keywords", [])
    all_signals = signal_keywords + extra_signals

    # Group signal keywords into batches of 3 for OR-combined queries
    signal_batches = []
    for i in range(0, len(all_signals), 3):
        batch = all_signals[i:i + 3]
        signal_batches.append(batch)

    for niche_kw in niche_kws[:3]:  # Top 3 niche keywords
        for source in sources:
            site_filter = source_to_site_filter(source)
            for batch in signal_batches[:3]:  # Top 3 signal batches
                # Build query: site filter + niche keyword + signal keywords (OR'd)
                signal_part = " OR ".join(f'"{s}"' for s in batch)
                query = f"{site_filter} {niche_kw} ({signal_part})"
                queries.append({
                    "query": query.strip(),
                    "niche": niche["name"],
                    "source": source,
                    "niche_keyword": niche_kw,
                    "signal_keywords": batch,
                })

                if len(queries) >= max_queries:
                    return queries

    return queries


def generate_quick_queries(topic):
    """Generate queries for a quick scan without full config."""
    sources = [
        "site:reddit.com",
        "site:news.ycombinator.com",
    ]
    signal_batches = [
        ['"I wish"', '"someone should build"', '"looking for"'],
        ['"frustrated"', '"alternative to"', '"anyone solved"'],
        ['"workaround"', '"need a tool"', '"why is there no"'],
    ]

    queries = []
    for source in sources:
        for batch in signal_batches:
            signal_part = " OR ".join(batch)
            query = f"{source} {topic} ({signal_part})"
            queries.append({
                "query": query.strip(),
                "niche": topic,
                "source": source,
                "niche_keyword": topic,
                "signal_keywords": [s.strip('"') for s in batch],
            })

    return queries


def generate_queries(config, niche_filter=None):
    """Generate all search queries from config."""
    signal_keywords = config.get("default_signal_keywords", DEFAULT_SIGNAL_KEYWORDS)
    max_per_scan = config.get("max_queries_per_scan", 30)
    all_queries = []

    for niche in config.get("niches", []):
        if niche_filter and niche["name"].lower() != niche_filter.lower():
            continue
        per_niche_max = max_per_scan // max(len(config.get("niches", [])), 1)
        queries = generate_queries_for_niche(niche, signal_keywords, per_niche_max)
        all_queries.extend(queries)

    return all_queries[:max_per_scan]


def ingest_results(results_path):
    """Parse raw web_search results into standardized findings.

    Expected input: JSON array of search result objects, each with at minimum:
    - title (str)
    - url (str)
    - description/snippet (str)

    Can also handle the nested format where results are grouped by query.
    """
    if results_path == "-":
        raw = json.load(sys.stdin)
    else:
        with open(results_path, "r") as f:
            raw = json.load(f)

    findings = []
    seen_urls = set()

    # Handle both flat array and grouped-by-query format
    items = []
    if isinstance(raw, list):
        for entry in raw:
            if isinstance(entry, dict) and "results" in entry:
                # Grouped format: [{query: ..., results: [...]}, ...]
                query_meta = {
                    "niche": entry.get("niche", ""),
                    "source": entry.get("source", ""),
                    "signal_keywords": entry.get("signal_keywords", []),
                }
                for r in entry["results"]:
                    items.append({**r, **query_meta})
            else:
                items.append(entry)
    elif isinstance(raw, dict) and "results" in raw:
        items = raw["results"]

    for item in items:
        url = item.get("url", "")
        if not url or url in seen_urls:
            continue
        seen_urls.add(url)

        title = item.get("title", "")
        snippet = item.get("description", item.get("snippet", ""))

        # Detect source from URL
        source = "unknown"
        if "reddit.com" in url:
            source = "reddit"
            # Extract subreddit
            parts = url.split("/r/")
            if len(parts) > 1:
                sub = parts[1].split("/")[0]
                source = f"reddit:r/{sub}"
        elif "news.ycombinator.com" in url:
            source = "hackernews"
        elif "github.com" in url:
            source = "github"

        # Match signal keywords found in title + snippet
        text = f"{title} {snippet}".lower()
        matched = []
        all_signals = item.get("signal_keywords", DEFAULT_SIGNAL_KEYWORDS)
        for kw in all_signals:
            if kw.lower() in text:
                matched.append(kw)

        finding = {
            "title": title,
            "url": url,
            "snippet": snippet,
            "source": item.get("source", source),
            "niche": item.get("niche", ""),
            "date": item.get("date", item.get("age", datetime.now().strftime("%Y-%m-%d"))),
            "matched_keywords": matched,
        }
        findings.append(finding)

    # Sort by number of matched keywords descending (rough signal proxy)
    findings.sort(key=lambda x: len(x["matched_keywords"]), reverse=True)

    return findings


def main():
    parser = argparse.ArgumentParser(description="Scan sources for opportunity signals")
    parser.add_argument("--generate-queries", action="store_true", help="Output search queries")
    parser.add_argument("--niche", help="Filter to a specific niche")
    parser.add_argument("--quick", metavar="TOPIC", help="Quick scan queries for a topic")
    parser.add_argument("--ingest", metavar="FILE", help="Ingest search results (- for stdin)")

    args = parser.parse_args()

    if args.quick:
        queries = generate_quick_queries(args.quick)
        print(json.dumps(queries, indent=2))
        return

    if args.generate_queries:
        config = load_config()
        queries = generate_queries(config, args.niche)
        print(json.dumps(queries, indent=2))
        return

    if args.ingest:
        findings = ingest_results(args.ingest)
        print(json.dumps(findings, indent=2))
        return

    parser.print_help()


if __name__ == "__main__":
    main()


path: scripts/score_signals.py
content: #!/usr/bin/env python3
"""Score and rank opportunity findings by signal strength, engagement,
freshness, competition, and recurrence.

Usage:
    score_signals.py <findings.json>           Score findings, output ranked JSON
    score_signals.py <findings.json> --config <config.json>   Custom weights
    score_signals.py - < findings.json         Read from stdin
"""

import argparse
import json
import math
import os
import re
import sys
from datetime import datetime, timedelta

SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
CONFIG_PATH = os.path.join(SKILL_DIR, "config.json")
HISTORY_PATH = os.path.join(SKILL_DIR, "history.json")

DEFAULT_WEIGHTS = {
    "signal_strength": 0.30,
    "engagement": 0.20,
    "freshness": 0.20,
    "competition": 0.15,
    "recurrence": 0.15,
}

# Strong demand signal keywords (weighted higher)
# Use flexible patterns — people don't say things exactly the same way
STRONG_SIGNALS = [
    "i wish", "someone should build", "someone would build", "someone needs to build",
    "shut up and take my money", "take my money",
    "i'd pay", "i would pay", "worth paying", "gladly pay",
    "why is there no", "why isn't there", "why does no one",
    "need a tool", "need something that",
]

# Moderate demand signal keywords
MODERATE_SIGNALS = [
    "frustrated", "so frustrated", "fed up", "drives me crazy", "annoyed",
    "looking for", "alternative to", "anyone solved", "has anyone solved",
    "workaround", "has anyone found", "hate that", "cobbled together",
    "duct tape", "jerry-rigged", "hack", "too expensive",
    "overkill", "bloated", "clunky", "terrible experience",
]

# Competition indicators (things that suggest the need IS being met)
COMPETITION_INDICATORS = [
    "try this", "I use", "check out", "have you tried", "I recommend",
    "we built", "just released", "solved this with",
]


def score_signal_strength(finding):
    """Score 0-10 based on how clearly the finding expresses unmet demand.

    Factors: number and type of matched keywords, keyword density in text.
    """
    matched = [k.lower() for k in finding.get("matched_keywords", [])]
    text = f"{finding.get('title', '')} {finding.get('snippet', '')}".lower()

    score = 0.0

    # Strong signal matches (3 points each, max 9)
    strong_matches = sum(1 for s in STRONG_SIGNALS if s in text)
    score += min(strong_matches * 3.0, 9.0)

    # Moderate signal matches (1.5 points each, max 6)
    moderate_matches = sum(1 for s in MODERATE_SIGNALS if s in text)
    score += min(moderate_matches * 1.5, 6.0)

    # Bonus for question marks (seeking behavior)
    question_marks = text.count("?")
    score += min(question_marks * 0.5, 1.5)

    # Bonus for specificity indicators (dollar amounts, named tools, metrics)
    if re.search(r'\$\d+', text):
        score += 1.0
    if re.search(r'\d+\s*(users|customers|employees|people|clients)', text):
        score += 0.5

    # Penalty for very short snippets (less context = less confidence)
    if len(text) < 50:
        score *= 0.5

    return min(round(score, 1), 10.0)


def score_engagement(finding):
    """Score 0-10 based on engagement signals extracted from snippet.

    Look for upvote counts, comment counts, and other engagement markers.
    """
    text = f"{finding.get('title', '')} {finding.get('snippet', '')}".lower()
    score = 5.0  # Default middle score when engagement data is absent

    # Try to extract upvote/point counts
    upvote_match = re.search(r'(\d+)\s*(upvotes?|points?|votes?|likes?)', text)
    if upvote_match:
        count = int(upvote_match.group(1))
        if count >= 100:
            score = 9.0
        elif count >= 50:
            score = 8.0
        elif count >= 20:
            score = 7.0
        elif count >= 10:
            score = 6.0
        elif count >= 5:
            score = 5.0
        else:
            score = 3.0

    # Try to extract comment counts
    comment_match = re.search(r'(\d+)\s*(comments?|replies?|responses?)', text)
    if comment_match:
        count = int(comment_match.group(1))
        if count >= 50:
            score = max(score, 9.0)
        elif count >= 20:
            score = max(score, 7.5)
        elif count >= 10:
            score = max(score, 6.5)

    return min(round(score, 1), 10.0)


def score_freshness(finding):
    """Score 0-10 based on how recent the finding is.

    Newer = more actionable = higher score.
    """
    date_str = finding.get("date", "")
    if not date_str:
        return 5.0  # Unknown age, middle score

    now = datetime.now()

    # Try parsing various date formats
    for fmt in ("%Y-%m-%d", "%Y-%m-%dT%H:%M:%S", "%b %d, %Y", "%B %d, %Y"):
        try:
            post_date = datetime.strptime(date_str.strip()[:19], fmt)
            days_old = (now - post_date).days

            if days_old <= 1:
                return 10.0
            elif days_old <= 7:
                return 9.0
            elif days_old <= 14:
                return 8.0
            elif days_old <= 30:
                return 7.0
            elif days_old <= 60:
                return 5.0
            elif days_old <= 90:
                return 3.0
            elif days_old <= 180:
                return 2.0
            else:
                return 1.0
        except ValueError:
            continue

    # Handle relative dates like "2 days ago", "3 hours ago"
    relative = re.search(r'(\d+)\s*(hour|day|week|month|year)s?\s*ago', date_str.lower())
    if relative:
        num = int(relative.group(1))
        unit = relative.group(2)
        if unit == "hour":
            return 10.0
        elif unit == "day":
            if num <= 1:
                return 10.0
            elif num <= 7:
                return 9.0
            elif num <= 14:
                return 8.0
            elif num <= 30:
                return 7.0
            else:
                return 5.0
        elif unit == "week":
            if num <= 1:
                return 8.5
            elif num <= 4:
                return 7.0
            else:
                return 5.0
        elif unit == "month":
            if num <= 1:
                return 6.0
            elif num <= 3:
                return 4.0
            else:
                return 2.0
        elif unit == "year":
            return 1.0

    return 5.0  # Can't parse, assume moderate


def score_competition(finding):
    """Score 0-10 where HIGH score = LOW competition (bigger opportunity).

    Check if the snippet suggests solutions already exist.
    """
    text = f"{finding.get('title', '')} {finding.get('snippet', '')}".lower()

    # Count competition indicators
    comp_count = sum(1 for c in COMPETITION_INDICATORS if c in text)

    if comp_count == 0:
        return 9.0  # No solutions mentioned — wide open
    elif comp_count == 1:
        return 7.0  # One mention — might be a partial solution
    elif comp_count == 2:
        return 5.0  # Multiple solutions mentioned
    elif comp_count <= 4:
        return 3.0  # Well-served market
    else:
        return 1.0  # Saturated

    return 5.0


def score_recurrence(finding):
    """Score 0-10 based on whether this signal has appeared in past scans.

    Reads from history.json to check for recurring topics.
    """
    if not os.path.exists(HISTORY_PATH):
        return 5.0  # No history yet, neutral score

    try:
        with open(HISTORY_PATH, "r") as f:
            history = json.load(f)
    except (json.JSONDecodeError, IOError):
        return 5.0

    # Normalize title for matching
    title_lower = finding.get("title", "").lower().strip()
    url = finding.get("url", "")

    signals = history.get("signals", {})

    # Check for URL match (exact same post recurring in searches)
    for sig_key, sig_data in signals.items():
        if url and url in sig_data.get("urls", []):
            appearances = len(sig_data.get("scan_dates", []))
            if appearances >= 5:
                return 10.0
            elif appearances >= 3:
                return 8.5
            elif appearances >= 2:
                return 7.0
            else:
                return 5.5

    # Check for topic similarity (rough title overlap)
    title_words = set(title_lower.split()) - {"the", "a", "an", "is", "are", "was",
                                                "for", "to", "in", "on", "of", "and",
                                                "or", "but", "with", "how", "what",
                                                "why", "has", "anyone", "does", "do"}
    for sig_key, sig_data in signals.items():
        key_words = set(sig_key.split())
        overlap = title_words & key_words
        if len(overlap) >= 3:
            appearances = len(sig_data.get("scan_dates", []))
            if appearances >= 3:
                return 8.0
            elif appearances >= 2:
                return 6.5

    return 5.0  # No match found, neutral


def compute_composite_score(scores, weights):
    """Compute weighted composite score from individual dimension scores."""
    composite = 0.0
    for key, weight in weights.items():
        composite += scores.get(key, 5.0) * weight
    return round(composite, 2)


def score_findings(findings, weights=None):
    """Score all findings and return sorted by composite score."""
    if weights is None:
        weights = DEFAULT_WEIGHTS

    scored = []
    for finding in findings:
        scores = {
            "signal_strength": score_signal_strength(finding),
            "engagement": score_engagement(finding),
            "freshness": score_freshness(finding),
            "competition": score_competition(finding),
            "recurrence": score_recurrence(finding),
        }
        composite = compute_composite_score(scores, weights)

        scored_finding = {
            **finding,
            "scores": scores,
            "composite_score": composite,
        }
        scored.append(scored_finding)

    # Sort by composite score descending
    scored.sort(key=lambda x: x["composite_score"], reverse=True)

    return scored


def main():
    parser = argparse.ArgumentParser(description="Score opportunity signal findings")
    parser.add_argument("findings", help="Path to findings JSON (or - for stdin)")
    parser.add_argument("--config", help="Path to config.json for custom weights")

    args = parser.parse_args()

    # Load findings
    if args.findings == "-":
        findings = json.load(sys.stdin)
    else:
        with open(args.findings, "r") as f:
            findings = json.load(f)

    # Load custom weights if provided
    weights = DEFAULT_WEIGHTS
    config_path = args.config or CONFIG_PATH
    if os.path.exists(config_path):
        try:
            with open(config_path, "r") as f:
                config = json.load(f)
            if "scoring_weights" in config:
                weights = config["scoring_weights"]
        except (json.JSONDecodeError, IOError):
            pass

    scored = score_findings(findings, weights)
    print(json.dumps(scored, indent=2))


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)