Skill: mvanhorn/last30days-v3-beta (version 3.0.0-open)

--- name: last30days version: "3.0.0-open" description: "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Perplexity, and more. AI agent scores by upvotes, likes, and real money. Also triggered by 'last30'." argument-hint: 'last30 AI video tools, last30 watch my competitor every week, last30 give me my briefing' allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch disable-model-invocation: true --- # last30days (open variant): Research + Watchlist + Briefings Multi-mode research skill with persistent knowledge accumulation. ## Command Routing Parse the user's first argument to determine the mode: | First word | Mode | Reference | |---|---|---| | `watch` | Watchlist management | `references/watchlist.md` | | `briefing` | Morning briefing | `references/briefing.md` | | `history` | Query accumulated knowledge | `references/history.md` | | *(anything else)* | One-shot research | `references/research.md` | ## Setup: Find Skill Root ```bash for dir in \ "." \ "${CLAUDE_PLUGIN_ROOT:-}" \ "${GEMINI_EXTENSION_DIR:-}" \ "$HOME/.gemini/extensions/last30days-skill" \ "$HOME/.gemini/extensions/last30days" \ "$HOME/.claude/skills/last30days" \ "$HOME/.agents/skills/last30days" \ "$HOME/.codex/skills/last30days"; do [ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break done if [ -z "${SKILL_ROOT:-}" ]; then echo "ERROR: Could not find scripts/last30days.py" >&2 exit 1 fi ``` Use `$SKILL_ROOT` for all script and reference file paths. ## Load Context At session start, read `${SKILL_ROOT}/variants/open/context.md` for user preferences and source quality notes. Update it after interactions. ## Shared Configuration - **Database**: `~/.local/share/last30days/research.db` (SQLite, WAL mode) - **Briefings**: `~/.local/share/last30days/briefs/` - **API keys**: `~/.config/last30days/.env` or environment variables - **Key priority**: env vars > config file ### API Keys | Key | Required | Purpose | |---|---|---| | `OPENAI_API_KEY` | For Reddit | Reddit search via OpenAI responses API | | `XAI_API_KEY` | For X (fallback) | X search via xAI Grok API | | `OPENROUTER_API_KEY` | Optional | Perplexity Sonar Pro search + AI reasoning (planning/reranking). Add `INCLUDE_SOURCES=perplexity` to enable. Use `--deep-research` for exhaustive reports. | | `PARALLEL_API_KEY` | Optional | Web search via Parallel AI | | `BRAVE_API_KEY` | Optional | Web search via Brave Search | Bird CLI provides free X search if installed. YouTube search uses yt-dlp (free). Run `python3 "${SKILL_ROOT}/scripts/last30days.py" --diagnose` to check source availability. ## Routing Logic After determining the mode, **read the corresponding reference file** using the Read tool: ``` Read: ${SKILL_ROOT}/variants/open/references/{mode}.md ``` Then follow the instructions in that reference file exactly.

path: CHANGELOG.md
content: # Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [3.0.0] - 2026-04

### Highlights

Intelligent search, fun judge, cross-source cluster merging, single-pass comparisons, and OpenClaw as a first-class citizen. The v3 engine doesn't just search for your topic -- it figures out *where* to search before the search begins. Engine architecture by @j-sperling.

### Added

- **Intelligent pre-research** -- Resolves X handles, subreddits, TikTok hashtags, and YouTube channels via a new Python brain before any API calls fire. Bidirectional: person to company, product to founder.
- **Fun judge / Best Takes** -- Second parallel LLM judge scores humor, cleverness, and virality. Surfaces the best reactions in a dedicated output section.
- **Cross-source cluster merging** -- Entity-based overlap detection merges the same story across Reddit, X, YouTube into one cluster instead of three separate items.
- **Single-pass comparisons** -- "X vs Y" runs one pass with entity-aware subqueries instead of three serial passes. 3 minutes instead of 12+.
- **GitHub as a source** -- Stars, reactions, and comments from repos and issues.
- **OpenClaw first-class citizen** -- Auto-resolve for engine-side pre-research. Device auth for frictionless ScrapeCreators signup.
- **Per-author cap** -- Max 3 items per author prevents single-voice dominance.
- **Entity disambiguation** -- Synthesis trusts resolved handles over keyword matches.
- **Perplexity Sonar Pro as additive source** -- AI-synthesized research with citations via OpenRouter. Opt-in via `INCLUDE_SOURCES=perplexity`. Returns structured narratives that complement social data.
- **Perplexity Deep Research** -- `--deep-research` flag for exhaustive 50+ citation reports (~$0.90/query). Premium opt-in for serious investigation.
- **OpenRouter as reasoning provider** -- One OPENROUTER_API_KEY powers planning, reranking, and Perplexity search. Auto-detected after Gemini/OpenAI/xAI.
- **Parallel AI grounding backend** -- `--web-backend parallel` or auto-detected via PARALLEL_API_KEY.
- **Grounding in planner** -- Grounding source properly registered in SOURCE_CAPABILITIES instead of force-injected.

### Changed

- YouTube transcript candidate pool widened 3x past music videos to reach talk/review content with captions
- Reddit comment enrichment sorted by total engagement (upvotes + comments), not just upvotes
- Polymarket display shows % odds only; dollar volumes removed
- 852 tests passing

### Contributors

- @j-sperling -- v3 engine architecture, Python pre-research brain
- @hnshah -- Watchlist features

## [2.9.4] - 2026-03-06

### Changed

- Move save into Python script via `--save-dir` flag - raw research data saved during the existing script Bash call, zero extra tool calls after invitation
- Remove entire "Save Research to Documents" section from SKILL.md (~45 lines removed)
- No more `📎` footer, no Bash heredoc, no `(No output)`, no multi-minute cogitation after research

## [2.9.3] - 2026-03-06

### Fixed

- **Critical:** Switch save from `run_in_background` to foreground Bash - background callbacks caused model to re-engage, hallucinate fake user messages, and generate unsolicited multi-paragraph responses
- Save uses foreground `cat >` heredoc (executes sub-second, no callback, no delayed notification)

## [2.9.2] - 2026-03-06

### Fixed

- Save research silently using background Bash heredoc instead of Write tool (eliminates "Wrote N lines..." clutter)
- Suppress follow-up text after background save completes (no more "Research briefing saved..." noise)
- Add `📎` footer line for save path instead of verbose confirmation

## [2.9.1] - 2026-03-05

### Highlights

Auto-save research briefings to `~/Documents/Last30Days/` as topic-named .md files. Every run now builds a personal research library automatically - no more manual copy-paste.

### Added

- Auto-save complete research briefings (synthesis, stats, follow-up suggestions) to `~/Documents/Last30Days/{topic-slug}.md` after every run
- Kebab-case filename generation from topic (e.g., "Claude Code skills" -> `claude-code-skills.md`)
- Duplicate topic handling: appends date suffix instead of overwriting (e.g., `claude-code-skills-2026-03-05.md`)
- Agent mode (`--agent`) also saves research files
- Brief confirmation after save: "Saved to ~/Documents/Last30Days/{slug}.md"

### Credits

- [@devin_explores](https://x.com/devin_explores) -- Inspired this feature by sharing their workflow of saving every last30days run into organized .md files ([PR #51](https://github.com/mvanhorn/last30days-skill/pull/51))

## [2.9.0] - 2026-03-05

### Highlights

ScrapeCreators Reddit as the default backend (one `SCRAPECREATORS_API_KEY` covers Reddit + TikTok + Instagram), smart subreddit discovery with relevance-weighted scoring, and top comments elevated with 10% scoring weight and prominent display.

### Added

- ScrapeCreators Reddit backend (`scripts/lib/reddit.py`) — keyword search, subreddit discovery, comment enrichment, all via `api.scrapecreators.com`
- Smart subreddit discovery with relevance-weighted scoring: frequency × recency × topic-word match, replacing pure frequency count
- `UTILITY_SUBS` blocklist to filter noise subreddits (r/tipofmytongue, r/whatisthisthing, etc.) from discovery results
- Top comment scoring: 10% weight in engagement formula via `log1p(top_comment_score)`
- Top comment rendering: `💬 Top comment` lines with upvote counts in compact and full report output
- Comment excerpt length increased from 300 → 400 chars; `comment_insights` limit raised from 7 → 10

### Changed

- `primaryEnv` switched from `OPENAI_API_KEY` to `SCRAPECREATORS_API_KEY` — one key now powers Reddit, TikTok, and Instagram
- Reddit engagement scoring formula: `0.55/0.40/0.05` (score/comments/ratio) → `0.50/0.35/0.05/0.10` (score/comments/ratio/top-comment)
- SKILL.md synthesis instructions updated to emphasize quoting top comments

### Fixed

- Utility subreddit noise in discovery (e.g., r/tipofmytongue appearing for unrelated topics)
- Reddit search no longer requires `OPENAI_API_KEY` — ScrapeCreators API handles search directly

## [2.8.0] - 2026-03-04

### Highlights

Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements. One API key (`SCRAPECREATORS_API_KEY`) now covers both TikTok and Instagram.

### Added

- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction (`scripts/lib/instagram.py`)
- `InstagramItem` dataclass, normalization, scoring (45% relevance / 25% recency / 30% engagement), deduplication, cross-source linking, and rendering
- Instagram in SKILL.md: stats template (`📸 Instagram:`), citation priority, item format description, output footer
- URL-to-name extraction examples in SKILL.md for cleaner web source display
- `--search=instagram` flag support

### Changed

- TikTok backend migrated from Apify to ScrapeCreators API (`api.scrapecreators.com`)
- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
- SKILL.md version bumped to v2.8
- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
- Security section updated: Apify → ScrapeCreators references

### Fixed

- Web stats line showing full URLs instead of plain domain names
- Trailing "Sources:" block appearing after skill invitation (WebSearch tool mandate conflict)
- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
- `$ARGUMENTS` quoting in SKILL.md for correct flag forwarding

## [2.1.0] - 2026-02-15

### Highlights

Three headline features: watchlists for always-on bots, YouTube transcripts as a 4th source, and Codex CLI compatibility. Plus bundled X search with no external CLI needed.

### Added

- Open-class skill with watchlists, briefings, and history modes (SQLite-backed, FTS5 full-text search, WAL mode) (`feat(open)`)
- YouTube as a 4th research source via yt-dlp -- search, view counts, and auto-generated transcript extraction (`feat: Add YouTube`)
- OpenAI Codex CLI compatibility -- install to `~/.agents/skills/last30days`, invoke with `$last30days` (`feat: Add Codex CLI`)
- Bundled X search -- vendored subset of Bird's Twitter GraphQL client (MIT, originally by @steipete), no external CLI needed (`v2.1: Bundle Bird X search`)
- Native web search backends: Parallel AI, Brave Search, OpenRouter/Perplexity Sonar Pro (`feat(engine)`)
- `--diagnose` flag for checking available sources and authentication status
- `--store` flag for SQLite accumulation (open variant)
- Conversational first-run experience (NUX) with dynamic source status (`feat(nux)`)

### Changed

- Smarter query construction -- strips noise words, auto-retries with shorter queries when X returns 0 results
- Two-phase search architecture -- Phase 1 discovers entities (@handles, r/subreddits), Phase 2 drills into them
- Reddit JSON enrichment -- real upvotes, comments, and upvote ratio from reddit.com/.json endpoint
- Engagement-weighted scoring: relevance 45%, recency 25%, engagement 30% (log1p dampening)
- Model auto-selection with 7-day cache and fallback chain (gpt-4.1 -> gpt-4o -> gpt-4o-mini)
- `--days=N` configurable lookback flag (thanks @jonthebeef, [#18](https://github.com/mvanhorn/last30days-skill/pull/18))
- Model fallback for unverified orgs (thanks @levineam, [#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- Marketplace plugin support via `.claude-plugin/plugin.json` (inspired by @galligan, [#1](https://github.com/mvanhorn/last30days-skill/pull/1))

### Fixed

- YouTube timeout increased to 90s, Reddit 429 rate limit fail-fast
- YouTube soft date filter -- keeps evergreen content instead of filtering to 0 results
- Eager import crash in `__init__.py` that broke Codex environments
- Reddit future timeout (same pattern as YouTube timeout bug)
- Process cleanup on timeout/kill -- tracks child PIDs for clean shutdown
- Windows Unicode fix for cp1252 emoji crash (thanks @JosephOIbrahim, [#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- X search returning 0 results on popular topics due to over-specific queries

### New Contributors

- @JosephOIbrahim -- Windows Unicode fix ([#17](https://github.com/mvanhorn/last30days-skill/pull/17))
- @levineam -- Model fallback for unverified orgs ([#16](https://github.com/mvanhorn/last30days-skill/pull/16))
- @jonthebeef -- `--days=N` configurable lookback ([#18](https://github.com/mvanhorn/last30days-skill/pull/18))

### Credits

- @steipete -- Bird CLI (vendored X search) and yt-dlp/summarize inspiration for YouTube transcripts
- @galligan -- Marketplace plugin inspiration
- @hutchins -- Pushed for YouTube feature

## [1.0.0] - 2026-01-15

Initial public release. Reddit + X search via OpenAI Responses API and xAI API.

[2.9.1]: https://github.com/mvanhorn/last30days-skill/compare/v2.9.0...v2.9.1
[2.9.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.8.0...v2.9.0
[2.8.0]: https://github.com/mvanhorn/last30days-skill/compare/v2.6.0...v2.8.0
[2.1.0]: https://github.com/mvanhorn/last30days-skill/compare/v1.0.0...v2.1.0
[1.0.0]: https://github.com/mvanhorn/last30days-skill/releases/tag/v1.0.0


path: context.md
content: # last30days Context

Agent memory for improving research quality over time.

## User Preferences
<!-- Record preferences discovered during interactions -->
<!-- e.g., "Prefers detailed technical analysis over general summaries" -->

## Source Quality Notes
<!-- Record which sources work best for which topics -->
<!-- e.g., "r/LocalLLaMA is highest signal for AI hardware topics" -->
<!-- e.g., "@kaboratech provides reliable AI tool reviews" -->

## Interaction History
<!-- Record topics researched and useful follow-up patterns -->
<!-- e.g., "2026-02-14: Researched 'AI video tools', user wanted Runway vs Kling comparison" -->


path: references/briefing.md
content: # Morning Briefing

Synthesize accumulated findings into a formatted briefing.

## Commands

| Command | Action |
|---|---|
| `briefing` | Generate today's briefing |
| `briefing --weekly` | Weekly digest with trends |
| `briefing --since YYYY-MM-DD` | Briefing since specific date |

## Generate Briefing

```bash
python3 "${SKILL_ROOT}/scripts/briefing.py" generate [--weekly] [--since DATE]
```

The script returns JSON with per-topic findings, staleness info, and cost data.

## Staleness Check

Before synthesizing, check each topic's freshness:
- **Fresh** (< 12h): show normally
- **Aging** (12-36h): note when last run was
- **Stale** (> 36h): warn user, suggest running `watch run-one "topic"`

## Daily Briefing Format

```
Good morning! Here's your research briefing for [DATE].

TL;DR: [One sentence about the top finding across all topics]

---

**[Topic 1]** (N new findings)
Top signal: [Highest engagement finding with source]
Also trending: [2nd finding], [3rd finding]

**[Topic 2]** (N new findings)
Top signal: [Highest engagement finding]
Also trending: [2nd finding]

---
Cost: $X.XX / $Y.YY budget | N topics active | N findings today
```

## Weekly Digest Format

```
Weekly digest for week of [DATE]:

**[Topic 1]**
This week: N findings (up/down X% from last week)
Trending up: [engagement increasing]
Key voices: @handle1, r/sub1

**[Topic 2]**
This week: N findings
Trending down: [engagement decreasing]
```

## Synthesis Rules

- Lead with people, not publications
- 3-5 topics max per briefing
- 2-3 findings per topic
- Include cost/budget footer
- Note any failed or stale topics

## No Data Handling

If no topics or no findings:
```
No briefing data available.

To get started:
1. Add a topic: /last30days watch add "your topic"
2. Run research: /last30days watch run-all
3. Generate briefing: /last30days briefing
```


path: references/history.md
content: # History & Knowledge Query

Query the accumulated findings database.

## Commands

| Command | Action |
|---|---|
| `history "topic"` | Show findings for a topic |
| `history "topic" --since=7d` | Findings from last N days |
| `history --search "query"` | Full-text search across all findings |
| `history --trending` | Topics with most recent activity |
| `history --stats` | Watchlist health dashboard |

## Topic History

```bash
python3 "${SKILL_ROOT}/scripts/store.py" query "TOPIC" [--since DAYS]
```

Display findings grouped by date (newest first):
```
**[Topic Name]** — N findings since [date]

[DATE]
- [Reddit] Title (score pts, N comments) — r/subreddit
- [X] Tweet text... (N likes) — @handle
- [YouTube] Video title (N views) — channel

[EARLIER DATE]
- ...
```

Mark updated findings (engagement changed since first seen).

## Full-Text Search

```bash
python3 "${SKILL_ROOT}/scripts/store.py" search "QUERY"
```

Uses FTS5 with BM25 ranking. Show results across all topics:
```
Search: "QUERY" — N results

1. [Reddit] Title — r/subreddit (topic: AI video)
   ...snippet with **highlighted** matches...

2. [X] Tweet text — @handle (topic: NVIDIA)
   ...snippet...
```

## Trending Topics

```bash
python3 "${SKILL_ROOT}/scripts/store.py" trending
```

Show topics ranked by recent activity:
```
Trending topics (last 7 days):

1. AI video tools — 12 new findings, engagement up 45%
2. NVIDIA news — 8 new findings, engagement steady
3. Claude Code — 3 new findings, engagement down 20%
```

## Stats Dashboard

```bash
python3 "${SKILL_ROOT}/scripts/store.py" stats
```

Display as a health dashboard:
```
Watchlist Health
- Active topics: N
- Total findings: N
- Database size: N KB

Research Runs (7 days)
- Successful: N
- Failed: N
- Cost: $X.XX

Source Breakdown
- Reddit: N findings
- X: N findings
- YouTube: N findings
- Web: N findings
```

## No Data Handling

If no findings exist:
```
No research history yet.

To start building knowledge:
1. Run research: /last30days "your topic"
2. Or add a watchlist topic: /last30days watch add "topic"
```


path: references/research.md
content: # One-Shot Research Mode (v3)

Research ANY topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.

---

## 1. Parse User Intent

Before doing anything, parse the user's input for:

1. **TOPIC**: What they want to learn about
2. **TARGET TOOL** (if specified): Where they'll use the prompts
3. **QUERY TYPE**:
   - **PROMPTING** - "X prompts", "prompting for X" -> copy-paste prompts
   - **RECOMMENDATIONS** - "best X", "top X" -> list of specific things
   - **NEWS** - "what's happening with X" -> current events
   - **COMPARISON** - "X vs Y", "X versus Y", "compare X and Y" -> side-by-side comparison
   - **GENERAL** - anything else -> broad understanding

Common patterns:
- `[topic] for [tool]` -> TOOL IS SPECIFIED
- `[topic] prompts for [tool]` -> TOOL IS SPECIFIED
- Just `[topic]` -> TOOL NOT SPECIFIED, that's OK
- "best [topic]" or "top [topic]" -> QUERY_TYPE = RECOMMENDATIONS
- "X vs Y" or "X versus Y" -> QUERY_TYPE = COMPARISON, TOPIC_A = X, TOPIC_B = Y

**Do NOT ask about target tool before research.** Run research first, ask after.

**Store these variables:**
- `TOPIC = [extracted topic]`
- `TARGET_TOOL = [extracted tool, or "unknown" if not specified]`
- `QUERY_TYPE = [PROMPTING | RECOMMENDATIONS | NEWS | COMPARISON | GENERAL]`
- `TOPIC_A = [first item]` (only if COMPARISON)
- `TOPIC_B = [second item]` (only if COMPARISON)

---

## 2. Confirm Topic

Display a branded one-liner before starting research. Build ACTIVE_SOURCES_LIST by checking what's configured in .env (Reddit, HN, Polymarket are always active; add X, YouTube, TikTok, Instagram, GitHub, Perplexity based on configured keys/tools).

For GENERAL / NEWS / RECOMMENDATIONS / PROMPTING queries:
```
/last30days - searching {ACTIVE_SOURCES_LIST} for what people are saying about {TOPIC}.
```

For COMPARISON queries:
```
/last30days - comparing {TOPIC_A} vs {TOPIC_B} across {ACTIVE_SOURCES_LIST}.
```

Do NOT show a multi-line "Parsed intent" block with TOPIC=, TARGET_TOOL=, QUERY_TYPE= variables. Do NOT promise a specific time. Do NOT list sources that aren't configured.

Then proceed immediately to research execution.

---

## 3. Handle / GitHub Resolution

**OpenClaw does not have WebSearch.** Skip manual handle resolution (Steps 0.5, 0.55, 0.75 from the main skill). Instead, add `--auto-resolve` to the research command. The engine will use configured web search backends (Brave, Exa, Serper) to discover subreddits, X handles, and context before planning.

If the user manually provides handles or community names, pass them through as CLI flags (see the flags list in Research Execution below). But do NOT attempt WebSearch-based resolution yourself.

---

## 4. Agent Mode (--agent flag)

If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):

1. **Skip** the intro display block
2. **Skip** any `AskUserQuestion` calls - use `TARGET_TOOL = "unknown"` if not specified
3. **Run** the research script exactly as normal
4. **Skip** the follow-up invitation
5. **Output** the complete research report and stop - do not wait for further input

Agent mode saves raw research data to `~/Documents/Last30Days/` automatically via `--save-dir`.

Agent mode report format:

```
## Research Report: {TOPIC}
Generated: {date} | Sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, Web

### Key Findings
[3-5 bullet points, highest-signal insights with citations]

### What I learned
{The full "What I learned" synthesis from normal output}

### Stats
{The standard stats block}
```

---

## 5. Comparison Mode (QUERY_TYPE = COMPARISON)

When the user asks "X vs Y", run ONE research pass with a comparison-optimized query that covers both entities AND their rivalry.

**Single pass with entity-aware subqueries:**
```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B}" --auto-resolve --emit=compact --save-dir=~/Documents/Last30Days --save-suffix=v3 --store 2>&1
```

If the user provided manual handles or subreddits, include those flags too.

Then skip the normal Research Execution below - go directly to the comparison synthesis format (see Synthesis section).

**Comparison output format:**

```
# {TOPIC_A} vs {TOPIC_B}: What the Community Says (Last 30 Days)

## Quick Verdict
[1-2 sentence data-driven summary: which one the community prefers and why, with source counts]

## {TOPIC_A}
Community Sentiment: [Positive/Mixed/Negative] ({N} mentions across {sources})

Strengths (what people love)
- [Point 1 with source attribution]
- [Point 2]

Weaknesses (common complaints)
- [Point 1 with source attribution]
- [Point 2]

## {TOPIC_B}
Community Sentiment: [Positive/Mixed/Negative] ({N} mentions across {sources})

Strengths (what people love)
- [Point 1 with source attribution]
- [Point 2]

Weaknesses (common complaints)
- [Point 1 with source attribution]
- [Point 2]

## Head-to-Head
[Synthesis from the combined search - what people say when directly comparing]

| Dimension | {TOPIC_A} | {TOPIC_B} |
|-----------|-----------|-----------|
| [Key dimension 1] | [A's position] | [B's position] |
| [Key dimension 2] | [A's position] | [B's position] |
| [Key dimension 3] | [A's position] | [B's position] |

## The Bottom Line
Choose {TOPIC_A} if... Choose {TOPIC_B} if... (based on actual community data, not assumptions)
```

Then show combined stats and the standard invitation section.

---

## 6. Research Execution

**Run the research script in the FOREGROUND with a 5-minute timeout.**

```bash
python3 "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --auto-resolve --emit=compact --save-dir=~/Documents/Last30Days --save-suffix=v3 --store 2>&1
```

Use a **timeout of 300000** (5 minutes). The `--store` flag persists findings for watchlist/briefing integration.

**Always include `--auto-resolve`** since OpenClaw has no WebSearch. The engine will use configured web search backends (Brave, Exa, Serper) to discover subreddits, X handles, and current events context before planning.

**Available flags** (pass through if the user provides them manually):
- `--x-handle={handle}` - primary X/Twitter handle (without @)
- `--x-related={handle1},{handle2}` - related X handles (comma-separated, without @)
- `--subreddits={sub1},{sub2}` - target subreddits (comma-separated, no r/ prefix)
- `--tiktok-hashtags={tag1},{tag2}` - TikTok hashtags to search
- `--tiktok-creators={creator1},{creator2}` - TikTok creator handles
- `--ig-creators={creator1},{creator2}` - Instagram creator handles
- `--github-user={username}` - GitHub username for person-mode search
- `--github-repo={owner/repo}` - GitHub repo for project-mode search (comma-separated for multiple)
- `--deep-research` - Perplexity Deep Research mode (exhaustive 50+ citation reports, ~$0.90/query, requires OPENROUTER_API_KEY + `INCLUDE_SOURCES=perplexity`)
- `--days=N` - look back N days instead of 30
- `--quick` - faster, fewer sources (8-12 each)
- `--deep` - comprehensive (50-70 Reddit, 40-60 X)

**Read the ENTIRE output.** It contains data sections for: Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and Web. If you miss sections, you will produce incomplete stats.

---

## 7. WebSearch Supplemental

**If your platform supports WebSearch**, use it after the script finishes to supplement with blogs, tutorials, and news.

Choose search queries based on QUERY_TYPE:

- **RECOMMENDATIONS**: `best {TOPIC} recommendations`, `{TOPIC} list examples`
- **NEWS**: `{TOPIC} news 2026`, `{TOPIC} announcement update`
- **PROMPTING**: `{TOPIC} prompts examples 2026`, `{TOPIC} techniques tips`
- **GENERAL**: `{TOPIC} 2026`, `{TOPIC} discussion`

Rules:
- **USE THE USER'S EXACT TERMINOLOGY**
- EXCLUDE reddit.com, x.com, twitter.com (covered by script)
- Do NOT output a separate "Sources:" block

**If your platform does NOT support WebSearch**, the `--auto-resolve` flag already provides web context via the engine's configured backends. Skip this step.

---

## 8. Synthesis / Judge Agent

### v3 Cluster-First Output

v3 returns results grouped by STORY/THEME (clusters), not by source. Each cluster represents one narrative thread found across multiple platforms.

**How to read v3 output:**
- `### 1. Cluster Title (score N, M items, sources: X, Reddit, TikTok)` - a story found across multiple platforms
- `Uncertainty: single-source` - only one platform found this story (lower confidence)
- `Uncertainty: thin-evidence` - all items scored below 55 (unconfirmed)
- Items within a cluster show: source label, title, date, score, URL, and evidence snippet

**Synthesis strategy for cluster-first output:**
1. **Synthesize per-cluster first.** Each cluster = one story. Summarize what each story is about.
2. **Multi-source clusters are highest confidence.** A cluster with items from Reddit + X + YouTube is much stronger than single-source.
3. **Check uncertainty tags.** "single-source" means treat with caution. "thin-evidence" means mention but caveat.
4. **Cross-cluster synthesis second.** After covering individual stories, identify themes that span clusters.
5. **Engagement signals still matter.** Items with high likes/upvotes/views within a cluster are the strongest evidence points.
6. **Quote directly from evidence snippets.** The snippets are pre-extracted best passages - use them.
7. Extract the top 3-5 actionable insights across all clusters.
8. **Disambiguation: trust your resolved entity.** When auto-resolve identified a specific entity, prioritize content about THAT entity. If results contain a different entity with the same name, lead with the one auto-resolve identified.

### Source-Specific Guidance (applies within clusters)

The Judge Agent must:
1. Weight Reddit/X sources HIGHER (they have engagement signals: upvotes, likes)
2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
3. Weight TikTok sources HIGH (they have views, likes, and caption content - viral signal)
4. Weight Instagram sources HIGH (influencer/creator perspective)
5. Weight WebSearch sources LOWER (no engagement data)
6. **For Reddit: Pay special attention to top comments** - they often contain the wittiest, most insightful, or funniest take. Quote them directly.
7. **For X: Reply clusters are gold.** When you see a cluster of replies to a recommendation-request tweet (someone asking "what's the best X?" and getting multiple independent responses), call this out prominently. This is the strongest form of community endorsement.
8. **For YouTube: Quote transcript highlights directly.** Attribute to the channel name.
9. **For TikTok: Note view counts and caption content.** Viral TikToks are strong cultural signal.
10. **For Instagram: Weight alongside TikTok.** Instagram provides unique creator/influencer perspective.
11. **For HN: Developer community signal.** Cite as "per HN" or "per hn/username."
12. **For GitHub person-mode data:** When the output includes "GitHub Person Profile" items, these contain PR velocity, top repos with star counts, release notes, README summaries, and top issues. Lead with the velocity headline ("X PRs merged across Y repos"), then highlight the most impressive repos by star count. Weave release notes into the narrative to show what actually shipped.
13. Identify patterns that appear across ALL sources (strongest signals)
14. Note any contradictions between sources
15. **Multi-source clusters (items from 3+ platforms) are the strongest signals.** Lead with these.

### Prediction Markets (Polymarket)

When Polymarket returns relevant markets, prediction market odds are among the highest-signal data points. Real money on outcomes cuts through opinion.

**6-point synthesis strategy:**

1. **Prefer structural/long-term markets over near-term deadlines.** Championship odds > regular season. Regime change > near-term strike deadline. IPO/major milestone > incremental update. When multiple markets exist, the bigger question is more interesting.

2. **When the topic is an outcome in a multi-outcome market, call out that specific outcome's odds and movement.** Don't just say "Polymarket has a market" - say "Arizona has a 28% chance, up 10% this month."

3. **Weave odds into the narrative as supporting evidence.** Don't isolate Polymarket data in its own paragraph. Instead: "Final Four buzz is building - Polymarket gives Arizona a 12% chance to win the championship (up 3% this week)."

4. **Citation format: show ONLY % odds. NEVER mention dollar volumes, liquidity, or betting amounts.** The % odds are the magic - the dollar figures are internal metrics that add zero value.

5. **When multiple relevant markets exist, highlight 3-5 of the most interesting ones** in your synthesis, ordered by importance (structural > near-term).

6. **Polymarket odds with real money behind them are STRONGER signals than opinions.** Always include specific percentages when markets are confirmed relevant.

### X Reply Cluster Weighting

When you see a cluster of replies to a recommendation-request tweet (someone asking "what's the best X?" and getting multiple independent responses), call this out prominently. This is the strongest form of community endorsement - real people independently making the same recommendation without coordination. Example: "In a thread where @ecom_cork asked for Loom alternatives, every reply said Tella."

### Fun Content

**If the research output includes a "## Best Takes" section or items tagged with `fun:` scores, weave at least 2-3 of the funniest/cleverest quotes into your synthesis.** Reddit comments and X posts with high fun scores are the voice of the people. A 1,338-upvote comment that says "Where's the limewire link" tells you more about the cultural moment than a news article. Quote the actual text. Don't put fun content in a separate section - mix it into the narrative where it fits naturally.

### ELI5 Mode

**If ELI5_MODE is true for this run, apply these writing guidelines to your ENTIRE synthesis. If false, skip this block and write normally.**

- Assume I know nothing about this topic. Zero context.
- No jargon without a quick explanation in parentheses
- Short sentences. One idea per sentence.
- Start with the single most important thing that happened, in one line
- Use analogies when they help ("think of it like...")
- Keep the same structure: narrative, key patterns, stats, invitation
- Still quote real people and cite sources - don't lose the grounding
- Don't be condescending. Simple is not stupid. ELI5 means accessible, not childish.

### Citation Priority Hierarchy

Most to least preferred:

1. @handles from X - "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit - "per r/subreddit" (prefer quoting top comments over thread titles)
3. YouTube channels - "per [channel name] on YouTube" (transcript-backed insights)
4. TikTok creators - "per @creator on TikTok" (viral/trending signal)
5. Instagram creators - "per @creator on Instagram" (influencer/creator signal)
6. HN discussions - "per HN" or "per hn/username" (developer community signal)
7. Polymarket - "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
8. Web sources - ONLY when other sources don't cover that specific fact

The tool's value is surfacing what PEOPLE are saying, not what journalists wrote. When both a web article and an X post cover the same fact, cite the X post.

### URL Formatting

NEVER paste raw URLs anywhere in the output - not in synthesis, not in stats, not in sources.
- **BAD:** "per https://www.rollingstone.com/music/..."
- **GOOD:** "per Rolling Stone"
- **BAD stats line:** `Web: 10 pages - https://later.com/blog/...`
- **GOOD stats line:** `Web: 10 pages - Later, Buffer, CNN, SocialBee`

Use the publication/site name, not the URL.

### Disambiguation

When auto-resolve or the research identified a specific entity (handles, subreddits, location context), prioritize content about THAT entity. If search results contain a different entity with the same name, lead with the entity your resolution identified. Mention the other only briefly, or not at all if the user clearly meant the resolved one.

### Anti-Pattern Warning

**Ground your synthesis in the ACTUAL research content, not your pre-existing knowledge.**

Read the research output carefully. Pay attention to:
- **Exact product/tool names** mentioned - don't conflate similar-sounding products
- **Specific quotes and insights** from the sources - use THESE, not generic knowledge
- **What the sources actually say**, not what you assume the topic is about

**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.

---

## 9. Display Results

### "What I learned" (format depends on QUERY_TYPE)

**If RECOMMENDATIONS** - show specific items with sources:
```
Most mentioned:

[Name] - {n}x mentions
Use Case: [what it does]
Sources: @handle1, r/sub, blog.com
```

**If PROMPTING/NEWS/GENERAL** - show synthesis:

Cite sparingly: 1-2 sources per topic in the intro, 1 source per pattern. Do NOT chain multiple citations.

```
What I learned:

{Topic 1} - [1-2 sentences about what people are saying, per @handle or r/sub]

{Topic 2} - [1-2 sentences, per @handle or r/sub]

KEY PATTERNS from the research:
1. [Pattern] - per @handle
2. [Pattern] - per r/sub
3. [Pattern] - per @handle
```

**If COMPARISON** - use the comparison output format from section 5 above.

### Stats Block

**Calculate actual totals from the research output.** Count posts/threads from each section. Sum engagement: parse likes, upvotes, views from each item. Identify top voices.

**Copy this EXACTLY, replacing only the {placeholders}:**

```
---
✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
├─ 🎵 TikTok: {N} videos │ {N} views │ {N} likes │ {N} with captions
├─ 📸 Instagram: {N} reels │ {N} views │ {N} likes │ {N} with captions
├─ 🧵 Threads: {N} posts │ {N} likes │ {N} replies
├─ 📌 Pinterest: {N} pins │ {N} saves │ {N} comments
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
├─ 🦋 Bluesky: {N} posts │ {N} likes │ {N} reposts
├─ 🇺🇸 Truth Social: {N} posts │ {N} likes │ {N} reposts
├─ 🐙 GitHub: {N} items │ {N} reactions │ {N} comments
├─ 📊 Polymarket: {N} markets │ {exact % odds}
├─ 🔮 Perplexity: {N} insights │ {N} citations
├─ 🌐 Web: {N} pages — Source Name, Source Name
├─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
└─ 📎 Raw results saved to ~/Documents/Last30Days/{slug}-raw.md
---
```

**CRITICAL: Omit any line with 0 results.** Do NOT show "0 threads", "0 stories", "0 markets", or "(no results this cycle)". If a source found nothing, DELETE that line entirely.

**Web line - how to extract site names from URLs:** Strip the protocol, path, and `www.` - use the recognizable publication name. List as comma-separated plain names.

---

## 10. Invitation

Adapt to QUERY_TYPE. Every invitation MUST include 2-3 specific example suggestions based on what you ACTUALLY learned from the research.

**If QUERY_TYPE = PROMPTING:**
```
---
I'm now an expert on {TOPIC} for {TARGET_TOOL}. What do you want to make? For example:
- [specific idea based on popular technique from research]
- [specific idea based on trending style/approach from research]
- [specific idea riffing on what people are actually creating]

Just describe your vision and I'll write a prompt you can paste straight into {TARGET_TOOL}.
```

**If QUERY_TYPE = RECOMMENDATIONS:**
```
---
I'm now an expert on {TOPIC}. Want me to go deeper? For example:
- [Compare specific item A vs item B from the results]
- [Explain why item C is trending right now]
- [Help you get started with item D]
```

**If QUERY_TYPE = NEWS:**
```
---
I'm now an expert on {TOPIC}. Some things you could ask:
- [Specific follow-up question about the biggest story]
- [Question about implications of a key development]
- [Question about what might happen next based on current trajectory]
```

**If QUERY_TYPE = COMPARISON:**
```
---
I've compared {TOPIC_A} vs {TOPIC_B} using the latest community data. Some things you could ask:
- [Deep dive into {TOPIC_A} alone with /last30days {TOPIC_A}]
- [Deep dive into {TOPIC_B} alone with /last30days {TOPIC_B}]
- [Focus on a specific dimension from the comparison table]
- [Look at a different time period with --days=7 or --days=90]
```

**If QUERY_TYPE = GENERAL:**
```
---
I'm now an expert on {TOPIC}. Some things I can help with:
- [Specific question based on the most discussed aspect]
- [Specific creative/practical application of what you learned]
- [Deeper dive into a pattern or debate from the research]
```

Context-aware: Only list sources that returned results in the closing line. Build the source list from your stats.

---

## 11. Follow-Up

After research, you are an **EXPERT** on this topic.

When the user responds, match their intent:

- **QUESTION** about the topic -> Answer from research (no new searches)
- **GO DEEPER** on a subtopic -> Elaborate from findings
- **CREATE/PROMPT** -> Write ONE prompt using research insights
- **Different topic** -> Run new research

### Mode Toggles

- **"eli5 on"** / **"eli5 mode"** / **"explain simpler"** -> Write `ELI5_MODE=true` to `~/.config/last30days/.env`. Confirm: "ELI5 mode on. All future runs will explain things like you're 5."
- **"eli5 off"** / **"normal mode"** / **"full detail"** -> Write `ELI5_MODE=false` to `~/.config/last30days/.env`. Confirm: "ELI5 mode off. Back to full detail."
- **"more fun"** / **"too serious"** -> Write `FUN_LEVEL=high` to `~/.config/last30days/.env`. Confirm: "Fun level set to high. Next run will surface more witty and viral content."
- **"less fun"** / **"too many jokes"** -> Write `FUN_LEVEL=low` to `~/.config/last30days/.env`. Confirm: "Fun level set to low. Next run will focus on the news."

### Writing Prompts

When the user wants a prompt, write a single, highly-tailored prompt using your research expertise.

**CRITICAL: Match the FORMAT the research recommends.** If research says to use JSON prompts with device specs, your prompt MUST be JSON. If research says structured params, use structured params.

Quality Checklist (run before delivering):
- FORMAT MATCHES RESEARCH - if research said JSON/structured/etc, prompt IS that format
- Directly addresses what the user said they want to create
- Uses specific patterns/keywords discovered in research
- Ready to paste with zero edits (or minimal [PLACEHOLDERS] clearly marked)
- Appropriate length and style for TARGET_TOOL

### Output Summary Footer (After Each Prompt)

```
---
Expert in: {TOPIC} for {TARGET_TOOL}
Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} TikTok videos ({sum} views) + {n} Instagram reels ({sum} views) + {n} HN stories ({sum} points) + {n} web pages

Want another prompt? Just tell me what you're creating next.
```

Only include sources with non-zero results in the footer.


path: references/watchlist.md
content: # Watchlist Management

Manage topics you want to track continuously. Findings accumulate in the SQLite database for briefings and history queries.

## Commands

| Command | Action |
|---|---|
| `watch add "topic"` | Add a topic (daily schedule) |
| `watch add "topic" --weekly` | Add with weekly schedule |
| `watch "topic"` | Shorthand for `watch add` |
| `watch remove "topic"` | Remove a topic |
| `watch list` | Show all topics with status |
| `watch config delivery [channel]` | Set delivery channel |
| `watch config budget AMOUNT` | Set daily cost budget |
| `watch run-all` | Run research for all topics now |
| `watch run-one "topic"` | Run research for one topic now |

## Adding a Topic

```bash
python3 "${SKILL_ROOT}/scripts/watchlist.py" add "TOPIC_NAME" [--weekly] [--queries "q1,q2"]
```

The script auto-bootstraps the SQLite database on first add.

**Default schedule**: Daily at 8am (`0 8 * * *`).
**Weekly**: Mondays at 8am (`0 8 * * 1`).

**After adding**, confirm to the user:
```
Added "TOPIC_NAME" to watchlist.
Schedule: daily at 8am (or weekly on Mondays)

To run research now: /last30days watch run-one "TOPIC_NAME"
To set up automated runs: add a cron/launchd job for `python3 ${SKILL_ROOT}/scripts/watchlist.py run-all`
```

## Removing a Topic

```bash
python3 "${SKILL_ROOT}/scripts/watchlist.py" remove "TOPIC_NAME"
```

Show confirmation or "not found" message.

## Listing Topics

```bash
python3 "${SKILL_ROOT}/scripts/watchlist.py" list
```

Display as a formatted table:
```
Topic         | Schedule     | Last Run     | Findings | Status
--------------+--------------+--------------+----------+--------
AI video      | daily 8am    | 2h ago       | 47       | ok
NVIDIA news   | weekly Mon   | 3d ago       | 23       | ok

Budget: $0.42 / $5.00 today
```

## Running Research

```bash
# All enabled topics (with budget guard)
python3 "${SKILL_ROOT}/scripts/watchlist.py" run-all

# Single topic
python3 "${SKILL_ROOT}/scripts/watchlist.py" run-one "TOPIC_NAME"
```

Show results: new findings count, updated findings, duration, and any errors.

## Configuration

```bash
# Set delivery channel (for future notification support)
python3 "${SKILL_ROOT}/scripts/watchlist.py" config delivery telegram

# Set daily budget limit
python3 "${SKILL_ROOT}/scripts/watchlist.py" config budget 10.00
```

## Scheduling

The watchlist doesn't auto-schedule. To automate, set up a system job:

**macOS (launchd)**:
```bash
# Run daily at 8am
crontab -e
# Add: 0 8 * * * python3 /path/to/scripts/watchlist.py run-all
```

**Linux (cron)**:
```bash
crontab -e
# Add: 0 8 * * * python3 /path/to/scripts/watchlist.py run-all
```

## Error Handling

- Duplicate topic: update the existing schedule
- Topic not found on remove: show "not found" message
- Budget exceeded: skip remaining topics, show which were skipped
- Research failure: record error, continue to next topic


path: scripts/briefing.py
content: #!/usr/bin/env python3
"""Morning briefing generator for last30days.

Synthesizes accumulated findings into formatted briefings.
The Python script collects the data; the agent (via SKILL.md) does the
beautiful synthesis. This script provides the structured data.

Usage:
    python3 briefing.py generate              # Daily briefing data
    python3 briefing.py generate --weekly     # Weekly digest data
    python3 briefing.py show [--date DATE]    # Show saved briefing
"""

import argparse
import json
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

import store

BRIEFS_DIR = Path.home() / ".local" / "share" / "last30days" / "briefs"


def _parse_sqlite_utc_timestamp(value: str) -> datetime:
    return datetime.strptime(value, "%Y-%m-%d %H:%M:%S").replace(tzinfo=timezone.utc)


def generate_daily(since: str = None) -> dict:
    """Generate daily briefing data.

    Returns structured data for the agent to synthesize into a beautiful briefing.
    """
    store.init_db()
    topics = store.list_topics()

    if not topics:
        return {
            "status": "no_topics",
            "message": "No watchlist topics yet. Add one with: last30days watch add \"your topic\"",
        }

    enabled = [t for t in topics if t["enabled"]]
    if not enabled:
        return {
            "status": "no_enabled",
            "message": "All topics are paused. Enable a topic to generate briefings.",
        }

    # Default: findings since yesterday
    if not since:
        since = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")

    briefing_topics = []
    total_new = 0

    for topic in enabled:
        findings = store.get_new_findings(topic["id"], since)
        last_run = topic.get("last_run")
        last_status = topic.get("last_status", "unknown")

        # Calculate staleness
        stale = False
        hours_ago = None
        if last_run:
            try:
                run_dt = _parse_sqlite_utc_timestamp(last_run)
                hours_ago = (datetime.now(timezone.utc) - run_dt).total_seconds() / 3600
                stale = hours_ago > 36  # Stale if > 36 hours
            except (ValueError, TypeError):
                stale = True

        topic_data = {
            "name": topic["name"],
            "findings": findings,
            "new_count": len(findings),
            "last_run": last_run,
            "last_status": last_status,
            "stale": stale,
            "hours_ago": round(hours_ago, 1) if hours_ago else None,
        }

        # Extract top finding by engagement
        if findings:
            top = max(findings, key=lambda f: f.get("engagement_score", 0))
            topic_data["top_finding"] = {
                "title": top.get("source_title", ""),
                "source": top.get("source", ""),
                "author": top.get("author", ""),
                "engagement": top.get("engagement_score", 0),
                "content": top.get("content", "")[:300],
            }

        briefing_topics.append(topic_data)
        total_new += len(findings)

    # Cost info
    daily_cost = store.get_daily_cost()
    budget = float(store.get_setting("daily_budget", "5.00"))

    # Find the single top finding across all topics (for TL;DR)
    all_findings = []
    for t in briefing_topics:
        for f in t["findings"]:
            f["_topic"] = t["name"]
            all_findings.append(f)

    top_overall = None
    if all_findings:
        top_overall = max(all_findings, key=lambda f: f.get("engagement_score", 0))

    result = {
        "status": "ok",
        "date": datetime.now().strftime("%Y-%m-%d"),
        "since": since,
        "topics": briefing_topics,
        "total_new": total_new,
        "total_topics": len(briefing_topics),
        "top_finding": {
            "title": top_overall.get("source_title", ""),
            "topic": top_overall.get("_topic", ""),
            "engagement": top_overall.get("engagement_score", 0),
        } if top_overall else None,
        "cost": {
            "daily": daily_cost,
            "budget": budget,
        },
        "failed_topics": [
            t["name"] for t in briefing_topics if t["last_status"] == "failed"
        ],
    }

    # Save briefing data
    _save_briefing(result)

    return result


def generate_weekly() -> dict:
    """Generate weekly digest data with trend analysis."""
    store.init_db()

    week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
    two_weeks_ago = (datetime.now() - timedelta(days=14)).strftime("%Y-%m-%d")

    topics = store.list_topics()
    if not topics:
        return {"status": "no_topics", "message": "No watchlist topics."}

    weekly_topics = []

    for topic in topics:
        if not topic["enabled"]:
            continue

        # This week's findings
        this_week = store.get_new_findings(topic["id"], week_ago)

        # Last week's findings (for comparison)
        conn = store._connect()
        try:
            last_week_rows = conn.execute(
                """SELECT * FROM findings
                   WHERE topic_id = ? AND first_seen >= ? AND first_seen < ? AND dismissed = 0
                   ORDER BY engagement_score DESC""",
                (topic["id"], two_weeks_ago, week_ago),
            ).fetchall()
            last_week = [dict(r) for r in last_week_rows]
        finally:
            conn.close()

        this_engagement = sum(f.get("engagement_score", 0) for f in this_week)
        last_engagement = sum(f.get("engagement_score", 0) for f in last_week)

        # Trend calculation
        if last_engagement > 0:
            engagement_change = ((this_engagement - last_engagement) / last_engagement) * 100
        else:
            engagement_change = 100 if this_engagement > 0 else 0

        weekly_topics.append({
            "name": topic["name"],
            "this_week_count": len(this_week),
            "last_week_count": len(last_week),
            "this_week_engagement": this_engagement,
            "last_week_engagement": last_engagement,
            "engagement_change_pct": round(engagement_change, 1),
            "top_findings": this_week[:5],  # Top 5 by engagement (already sorted)
        })

    result = {
        "status": "ok",
        "type": "weekly",
        "week_of": week_ago,
        "topics": weekly_topics,
    }

    _save_briefing(result, suffix="-weekly")

    return result


def show_briefing(date: str = None) -> dict:
    """Load a saved briefing by date."""
    if not date:
        date = datetime.now().strftime("%Y-%m-%d")

    path = BRIEFS_DIR / f"{date}.json"
    if not path.exists():
        # Try weekly
        path = BRIEFS_DIR / f"{date}-weekly.json"

    if not path.exists():
        return {"status": "not_found", "message": f"No briefing found for {date}."}

    with open(path) as f:
        return json.load(f)


def _save_briefing(data: dict, suffix: str = ""):
    """Save briefing data to local archive."""
    BRIEFS_DIR.mkdir(parents=True, exist_ok=True)
    date = datetime.now().strftime("%Y-%m-%d")
    path = BRIEFS_DIR / f"{date}{suffix}.json"
    with open(path, "w") as f:
        json.dump(data, f, indent=2, default=str)


def main():
    parser = argparse.ArgumentParser(description="Generate last30days briefings")
    sub = parser.add_subparsers(dest="command")

    # generate
    g = sub.add_parser("generate", help="Generate a briefing")
    g.add_argument("--weekly", action="store_true", help="Weekly digest")
    g.add_argument("--since", help="Findings since date (YYYY-MM-DD)")

    # show
    s = sub.add_parser("show", help="Show a saved briefing")
    s.add_argument("--date", help="Date (YYYY-MM-DD, default: today)")

    args = parser.parse_args()

    if args.command == "generate":
        if args.weekly:
            result = generate_weekly()
        else:
            result = generate_daily(since=args.since)
        print(json.dumps(result, indent=2, default=str))

    elif args.command == "show":
        result = show_briefing(date=args.date)
        print(json.dumps(result, indent=2, default=str))

    else:
        parser.print_help()
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/last30days.py
content: #!/usr/bin/env python3
# ruff: noqa: E402
"""last30days-3 v3.0.0-alpha CLI."""

from __future__ import annotations

import argparse
import atexit
import json
import os
import re
import signal
import sys
import threading
from pathlib import Path

SCRIPT_DIR = Path(__file__).parent.resolve()
sys.path.insert(0, str(SCRIPT_DIR))

from lib import env, pipeline, render, schema, ui

_child_pids: set[int] = set()
_child_pids_lock = threading.Lock()


def register_child_pid(pid: int) -> None:
    with _child_pids_lock:
        _child_pids.add(pid)


def unregister_child_pid(pid: int) -> None:
    with _child_pids_lock:
        _child_pids.discard(pid)


def _cleanup_children() -> None:
    with _child_pids_lock:
        pids = list(_child_pids)
    for pid in pids:
        try:
            os.killpg(os.getpgid(pid), signal.SIGTERM)
        except (ProcessLookupError, PermissionError, OSError):
            continue


atexit.register(_cleanup_children)


def parse_search_flag(raw: str) -> list[str]:
    sources = []
    for source in raw.split(","):
        source = source.strip().lower()
        if not source:
            continue
        normalized = pipeline.SEARCH_ALIAS.get(source, source)
        if normalized not in pipeline.MOCK_AVAILABLE_SOURCES:
            raise SystemExit(f"Unknown search source: {source}")
        if normalized not in sources:
            sources.append(normalized)
    if not sources:
        raise SystemExit("--search requires at least one source.")
    return sources


def slugify(value: str) -> str:
    slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
    return slug or "last30days"


def save_output(report: schema.Report, emit: str, save_dir: str, suffix: str = "") -> Path:
    from datetime import datetime
    path = Path(save_dir).expanduser().resolve()
    path.mkdir(parents=True, exist_ok=True)
    slug = slugify(report.topic)
    extension = "json" if emit == "json" else "md"
    suffix_part = f"-{suffix}" if suffix else ""
    out_path = path / f"{slug}-raw{suffix_part}.{extension}"
    if out_path.exists():
        out_path = path / f"{slug}-raw{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
    # Always save the FULL dump to disk (all items, all sources, transcripts).
    # Claude sees compact clusters via --emit=compact on stdout.
    # The saved file is the complete debug artifact.
    if emit == "json":
        content = emit_output(report, emit)
    else:
        content = render.render_full(report)
    out_path.write_text(content)
    return out_path


def emit_output(report: schema.Report, emit: str, fun_level: str = "medium") -> str:
    if emit == "json":
        return json.dumps(schema.to_dict(report), indent=2, sort_keys=True)
    if emit in {"compact", "md"}:
        return render.render_compact(report, fun_level=fun_level)
    if emit == "context":
        return render.render_context(report)
    raise SystemExit(f"Unsupported emit mode: {emit}")


def persist_report(report: schema.Report) -> dict[str, int]:
    import store

    store.init_db()
    topic_row = store.add_topic(report.topic)
    topic_id = topic_row["id"]
    source_mode = ",".join(sorted(report.items_by_source)) or "v3"
    run_id = store.record_run(topic_id, source_mode=source_mode, status="running")
    try:
        findings = store.findings_from_report(report)
        counts = store.store_findings(run_id, topic_id, findings)
        store.update_run(
            run_id,
            status="completed",
            findings_new=counts["new"],
            findings_updated=counts["updated"],
        )
        return counts
    except Exception as exc:
        store.update_run(run_id, status="failed", error_message=str(exc)[:500])
        raise


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Research a topic across live social, market, and grounded web sources.")
    parser.add_argument("topic", nargs="*", help="Research topic")
    parser.add_argument("--emit", default="compact", choices=["compact", "json", "context", "md"])
    parser.add_argument("--search", help="Comma-separated source list")
    parser.add_argument("--quick", action="store_true", help="Lower-latency retrieval profile")
    parser.add_argument("--deep", action="store_true", help="Higher-recall retrieval profile")
    parser.add_argument("--debug", action="store_true", help="Enable HTTP debug logging")
    parser.add_argument("--mock", action="store_true", help="Use mock retrieval fixtures")
    parser.add_argument("--diagnose", action="store_true", help="Print provider and source availability")
    parser.add_argument("--save-dir", help="Optional directory for saving the rendered output")
    parser.add_argument("--store", action="store_true", help="Persist ranked findings to the SQLite research store")
    parser.add_argument("--x-handle", help="X handle for targeted supplemental search")
    parser.add_argument("--x-related", help="Comma-separated related X handles (searched with lower weight)")
    parser.add_argument("--web-backend", default="auto",
                        choices=["auto", "brave", "exa", "serper", "parallel", "none"],
                        help="Web search backend (default: auto, tries Brave then Exa then Serper then Parallel)")
    parser.add_argument("--deep-research", action="store_true",
                        help="Use Perplexity Deep Research (~$0.90/query) for in-depth analysis. Requires OPENROUTER_API_KEY.")
    parser.add_argument("--plan", help="JSON query plan (skips internal LLM planner). Can be a JSON string or a file path.")
    parser.add_argument("--save-suffix", help="Suffix for saved output filename (e.g., 'gemini' → kanye-west-raw-gemini.md)")
    parser.add_argument("--subreddits", help="Comma-separated subreddit names to search (e.g., SaaS,Entrepreneur)")
    parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
    parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
    parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
    parser.add_argument("--lookback-days", type=int, default=30, help="Number of days to look back for research (default: 30, watchlist uses 90)")
    parser.add_argument("--auto-resolve", action="store_true",
                        help="Use web search to discover subreddits/handles before planning (for platforms without WebSearch)")
    parser.add_argument("--github-user", help="GitHub username for person-mode search (e.g., steipete)")
    parser.add_argument("--github-repo", help="Comma-separated owner/repo for project-mode search (e.g., openclaw/openclaw,paperclipai/paperclip)")
    return parser


def _missing_sources_for_promo(diag: dict[str, object]) -> str | None:
    available = set(diag.get("available_sources") or [])
    missing = []
    if "reddit" not in available:
        missing.append("reddit")
    if "x" not in available:
        missing.append("x")
    if "grounding" not in available:
        missing.append("web")
    if not missing:
        return None
    if "reddit" in missing and "x" in missing:
        return "both"
    return missing[0]


def _show_runtime_ui(report: schema.Report, progress: ui.ProgressDisplay, diag: dict[str, object]) -> None:
    counts = {source: len(items) for source, items in report.items_by_source.items()}
    display_sources = list(
        dict.fromkeys(
            [
                *report.query_plan.source_weights.keys(),
                *report.items_by_source.keys(),
                *report.errors_by_source.keys(),
            ]
        )
    )
    progress.end_processing()
    progress.show_complete(
        source_counts=counts,
        display_sources=display_sources,
    )
    promo = _missing_sources_for_promo(diag)
    if promo:
        progress.show_promo(promo, diag=diag)


def main() -> int:
    parser = build_parser()
    # Use parse_known_args so setup sub-flags (--device-auth, --github,
    # --openclaw) pass through without argparse hard-exiting.
    args, extra_argv = parser.parse_known_args()
    if args.debug:
        os.environ["LAST30DAYS_DEBUG"] = "1"

    config = env.get_config()

    # Handle setup subcommand
    topic = " ".join(args.topic).strip()
    if topic.lower() == "setup":
        from lib import setup_wizard
        if "--openclaw" in extra_argv:
            results = setup_wizard.run_openclaw_setup(config)
            print(json.dumps(results))
            return 0
        if "--github" in extra_argv:
            results = setup_wizard.run_github_auth()
            print(json.dumps(results))
            return 0
        if "--device-auth" in extra_argv:
            results = setup_wizard.run_full_device_auth()
            print(json.dumps(results))
            return 0
        sys.stderr.write("Running auto-setup...\n")
        results = setup_wizard.run_auto_setup(config)
        from_browser = "auto"
        if results.get("creds_found"):
            first_browser = next(iter(results["creds_found"].values()))
            from_browser = first_browser
        setup_wizard.write_setup_config(env.CONFIG_FILE, from_browser=from_browser)
        results["env_written"] = True
        sys.stderr.write(setup_wizard.get_setup_status_text(results) + "\n")
        return 0

    requested_sources = parse_search_flag(args.search) if args.search else None
    diag = pipeline.diagnose(config, requested_sources)

    if args.diagnose:
        print(json.dumps(diag, indent=2, sort_keys=True))
        return 0

    if not topic:
        parser.print_usage(sys.stderr)
        return 2

    progress = ui.ProgressDisplay(topic, show_banner=True)
    progress.start_processing()

    depth = "deep" if args.deep else "quick" if args.quick else "default"
    try:
        x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
        subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
        tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
        tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
        ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
        # Parse external plan if provided via --plan flag
        external_plan = None
        if args.plan:
            import json as _json
            plan_str = args.plan
            if os.path.isfile(plan_str):
                plan_str = open(plan_str).read()
            try:
                external_plan = _json.loads(plan_str)
            except _json.JSONDecodeError as exc:
                sys.stderr.write(f"[Planner] Invalid --plan JSON: {exc}\n")

        # Auto-resolve: use web search to discover subreddits/handles before planning.
        # This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
        # without WebSearch (OpenClaw, Codex, raw CLI).
        if args.auto_resolve and not external_plan:
            from lib import resolve
            resolution = resolve.auto_resolve(topic, config)
            if resolution.get("subreddits") and not subreddits:
                subreddits = resolution["subreddits"]
                sys.stderr.write(f"[AutoResolve] Subreddits: {', '.join(subreddits)}\n")
            if resolution.get("x_handle") and not args.x_handle:
                args.x_handle = resolution["x_handle"]
                sys.stderr.write(f"[AutoResolve] X handle: @{args.x_handle}\n")
            if resolution.get("github_user") and not args.github_user:
                args.github_user = resolution["github_user"]
                sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
            if resolution.get("github_repos") and not args.github_repo:
                args.github_repo = ",".join(resolution["github_repos"])
                sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
            if resolution.get("context"):
                # Inject context into external_plan metadata for the planner to use
                if not external_plan:
                    external_plan = None  # planner will use its own, but with context
                # Store context for the planner prompt injection
                config["_auto_resolve_context"] = resolution["context"]
                sys.stderr.write(f"[AutoResolve] Context: {resolution['context'][:80]}...\n")

        github_user = args.github_user.lstrip("@").lower() if args.github_user else None
        github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None

        # --deep-research: auto-enable perplexity source and set deep flag
        if args.deep_research:
            if not config.get("OPENROUTER_API_KEY"):
                print("Error: --deep-research requires OPENROUTER_API_KEY", file=sys.stderr)
                sys.exit(1)
            config["_deep_research"] = True
            # Auto-enable perplexity in INCLUDE_SOURCES
            include = config.get("INCLUDE_SOURCES") or ""
            if "perplexity" not in include.lower():
                config["INCLUDE_SOURCES"] = f"{include},perplexity" if include else "perplexity"

        report = pipeline.run(
            topic=topic,
            config=config,
            depth=depth,
            requested_sources=requested_sources,
            mock=args.mock,
            x_handle=args.x_handle,
            x_related=x_related,
            web_backend=args.web_backend,
            external_plan=external_plan,
            subreddits=subreddits,
            tiktok_hashtags=tiktok_hashtags,
            tiktok_creators=tiktok_creators,
            ig_creators=ig_creators,
            lookback_days=args.lookback_days,
            github_user=github_user,
            github_repos=github_repos,
        )
    except Exception as exc:
        progress.end_processing()
        progress.show_error(str(exc))
        raise
    _show_runtime_ui(report, progress, diag)
    if args.store:
        counts = persist_report(report)
        sys.stderr.write(
            f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
        )
        sys.stderr.flush()

    # Show quality nudge if applicable
    try:
        from lib import quality_nudge
        quality = quality_nudge.compute_quality_score(config, {})
        if quality.get("nudge_text"):
            sys.stderr.write(f"\n{quality['nudge_text']}\n")
            sys.stderr.flush()
    except Exception:
        pass

    fun_level = config.get("FUN_LEVEL", "medium").lower()
    rendered = emit_output(report, args.emit, fun_level=fun_level)
    if args.save_dir:
        save_path = save_output(report, args.emit, args.save_dir, suffix=args.save_suffix or "")
        sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
        sys.stderr.flush()
    print(rendered)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/lib/__init__.py
content: # last30days library modules


path: scripts/lib/bird_x.py
content: """Bird X search client for the v3.0.0 last30days pipeline.

Uses a vendored subset of @steipete/bird v0.8.0 (MIT License) to search X
via Twitter's GraphQL API. No external `bird` CLI binary needed - just Node.js.
See scripts/lib/vendor/bird-search/package.json for authoritative version.
"""

import json
import os
import signal
import shutil
import subprocess
import sys
from pathlib import Path

from . import http, log
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple

from .relevance import token_overlap_relevance as _compute_relevance


def _first_of(*values):
    """Return first value that is not None."""
    for v in values:
        if v is not None:
            return v
    return None

# Path to the vendored bird-search wrapper
_BIRD_SEARCH_MJS = Path(__file__).parent / "vendor" / "bird-search" / "bird-search.mjs"

# Depth configurations: number of results to request
DEPTH_CONFIG = {
    "quick": 12,
    "default": 30,
    "deep": 60,
}

# Module-level credentials injected from .env config
_credentials: Dict[str, str] = {}


def set_credentials(auth_token: Optional[str], ct0: Optional[str]):
    """Inject AUTH_TOKEN/CT0 from .env config so Node subprocesses can use them."""
    if auth_token:
        _credentials['AUTH_TOKEN'] = auth_token
    if ct0:
        _credentials['CT0'] = ct0


def _has_injected_credentials() -> bool:
    """Return True when both X credentials were injected from config."""
    return bool(_credentials.get('AUTH_TOKEN') and _credentials.get('CT0'))


def _has_process_credentials() -> bool:
    """Return True when AUTH_TOKEN/CT0 are present in process env."""
    return bool(os.environ.get("AUTH_TOKEN") and os.environ.get("CT0"))


def _subprocess_env() -> Dict[str, str]:
    """Build env dict for Node subprocesses, merging injected credentials."""
    env = os.environ.copy()
    env.update(_credentials)
    return env


def _log(msg: str):
    log.source_log("Bird", msg, tty_only=False)


def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for X search.

    X search is literal keyword AND matching — all words must appear.
    Aggressively strip question/meta/research words to keep only the
    core product/concept name (max 5 words).
    """
    from .query import extract_core_subject
    return extract_core_subject(topic, max_words=5, strip_suffixes=True)


def is_bird_installed() -> bool:
    """Check if vendored Bird search module is available.

    Returns:
        True if bird-search.mjs exists and Node.js is in PATH.
    """
    if not _BIRD_SEARCH_MJS.exists():
        return False
    return shutil.which("node") is not None


def is_bird_authenticated() -> Optional[str]:
    """Check if explicit X credentials are available.

    Returns:
        Auth source string if authenticated, None otherwise.
    """
    if not is_bird_installed():
        return None

    if _has_injected_credentials():
        return "env AUTH_TOKEN"
    if _has_process_credentials():
        return "env AUTH_TOKEN"
    return None


def check_npm_available() -> bool:
    """Check if npm is available (kept for API compatibility).

    Returns:
        True if 'npm' command is available in PATH, False otherwise.
    """
    return shutil.which("npm") is not None


def install_bird() -> Tuple[bool, str]:
    """No-op. Bird search is vendored in v3.0.0, no installation needed.

    Returns:
        Tuple of (success, message).
    """
    if is_bird_installed():
        return True, "Bird search is bundled with /last30days v3.0.0 - no installation needed."
    if not shutil.which("node"):
        return False, "Node.js 22+ is required for X search. Install Node.js first."
    return False, f"Vendored bird-search.mjs not found at {_BIRD_SEARCH_MJS}"


def get_bird_status() -> Dict[str, Any]:
    """Get comprehensive Bird search status.

    Returns:
        Dict with keys: installed, authenticated, username, can_install
    """
    installed = is_bird_installed()
    auth_source = is_bird_authenticated() if installed else None

    return {
        "installed": installed,
        "authenticated": auth_source is not None,
        "username": auth_source,  # Now returns auth source (e.g., "Safari", "env AUTH_TOKEN")
        "can_install": True,  # Always vendored in v3.0.0
    }


def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
    """Run a search using the vendored bird-search.mjs module.

    Args:
        query: Full search query string (including since: filter)
        count: Number of results to request
        timeout: Timeout in seconds

    Returns:
        Raw Bird JSON response or error dict.
    """
    cmd = [
        "node", str(_BIRD_SEARCH_MJS),
        query,
        "--count", str(count),
        "--json",
    ]

    # Use process groups for clean cleanup on timeout/kill
    preexec = os.setsid if hasattr(os, 'setsid') else None

    try:
        proc = subprocess.Popen(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            preexec_fn=preexec,
            env=_subprocess_env(),
        )

        # Register for cleanup tracking (if available)
        try:
            from last30days import register_child_pid, unregister_child_pid
            register_child_pid(proc.pid)
        except ImportError:
            pass

        try:
            stdout, stderr = proc.communicate(timeout=timeout)
        except subprocess.TimeoutExpired:
            # Kill the entire process group
            try:
                os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
            except (ProcessLookupError, PermissionError, OSError):
                proc.kill()
            proc.wait(timeout=5)
            return {"error": f"Search timed out after {timeout}s", "items": []}
        finally:
            try:
                from last30days import unregister_child_pid
                unregister_child_pid(proc.pid)
            except Exception:
                pass

        if proc.returncode != 0:
            error = stderr.strip() if stderr else "Bird search failed"
            return {"error": error, "items": []}

        output = stdout.strip() if stdout else ""
        if not output:
            return {"items": []}

        return json.loads(output)

    except json.JSONDecodeError as e:
        return {"error": f"Invalid JSON response: {e}", "items": []}
    except Exception as e:
        return {"error": str(e), "items": []}


def search_x(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
) -> Dict[str, Any]:
    """Search X using Bird CLI with automatic retry on 0 results.

    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD) - unused but kept for API compatibility
        depth: Research depth - "quick", "default", or "deep"

    Returns:
        Raw Bird JSON response or error dict.
    """
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    timeout = 30 if depth == "quick" else 45 if depth == "default" else 60

    # Extract core subject - X search is literal, not semantic
    core_topic = _extract_core_subject(topic)
    query = f"{core_topic} since:{from_date}"

    _log(f"Searching: {query}")
    response = _run_bird_search(query, count, timeout)

    # Check if we got results
    items = parse_bird_response(response, query=core_topic)

    # Retry with OR groups for multi-word queries (X supports OR operator)
    core_words = core_topic.split()
    if not items and len(core_words) >= 2:
        from .query import extract_compound_terms
        compounds = extract_compound_terms(topic)
        if compounds:
            # Build OR-group query: ("multi-agent" OR "agent simulation") since:DATE
            or_parts = ' OR '.join(f'"{t}"' for t in compounds[:3])
            _log(f"0 results for '{core_topic}', retrying with OR groups: {or_parts}")
            query = f"({or_parts}) since:{from_date}"
            response = _run_bird_search(query, count, timeout)
            items = parse_bird_response(response, query=core_topic)

    # Retry with fewer keywords if still 0 results and query has 3+ words
    if not items and len(core_words) > 2:
        shorter = ' '.join(core_words[:2])
        _log(f"0 results for '{core_topic}', retrying with '{shorter}'")
        query = f"{shorter} since:{from_date}"
        response = _run_bird_search(query, count, timeout)
        items = parse_bird_response(response, query=core_topic)

    # Last-chance retry: use strongest remaining token (often the product name)
    if not items and core_words:
        low_signal = {
            'trendiest', 'trending', 'hottest', 'hot', 'popular', 'viral',
            'best', 'top', 'latest', 'new', 'plugin', 'plugins',
            'skill', 'skills', 'tool', 'tools',
        }
        candidates = [w for w in core_words if w not in low_signal]
        if candidates:
            strongest = max(candidates, key=len)
            _log(f"0 results for '{core_topic}', retrying with strongest token '{strongest}'")
            query = f"{strongest} since:{from_date}"
            response = _run_bird_search(query, count, timeout)

    return response


def search_handles(
    handles: List[str],
    topic: Optional[str],
    from_date: str,
    count_per: int = 5,
) -> List[Dict[str, Any]]:
    """Search specific X handles for topic-related content.

    Runs targeted Bird searches using `from:handle topic` syntax.
    Used in Phase 2 supplemental search after entity extraction.

    Args:
        handles: List of X handles to search (without @)
        topic: Search topic (core subject), or None for unfiltered search
        from_date: Start date (YYYY-MM-DD)
        count_per: Results to request per handle

    Returns:
        List of raw item dicts (same format as parse_bird_response output).
    """
    all_items = []
    core_topic = _extract_core_subject(topic) if topic else None

    for handle in handles:
        handle = handle.lstrip("@")
        if core_topic:
            query = f"from:{handle} {core_topic} since:{from_date}"
        else:
            query = f"from:{handle} since:{from_date}"

        cmd = [
            "node", str(_BIRD_SEARCH_MJS),
            query,
            "--count", str(count_per),
            "--json",
        ]

        preexec = os.setsid if hasattr(os, 'setsid') else None

        try:
            proc = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                text=True,
                preexec_fn=preexec,
                env=_subprocess_env(),
            )

            try:
                stdout, stderr = proc.communicate(timeout=15)
            except subprocess.TimeoutExpired:
                try:
                    os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
                except (ProcessLookupError, PermissionError, OSError):
                    proc.kill()
                proc.wait(timeout=5)
                _log(f"Handle search timed out for @{handle}")
                continue

            if proc.returncode != 0:
                _log(f"Handle search failed for @{handle}: {(stderr or '').strip()}")
                continue

            output = (stdout or "").strip()
            if not output:
                continue

            response = json.loads(output)
            items = parse_bird_response(response, query=core_topic)
            all_items.extend(items)

        except json.JSONDecodeError:
            _log(f"Invalid JSON from handle search for @{handle}")
        except (OSError, subprocess.SubprocessError) as e:
            _log(f"Handle search error for @{handle}: {e}")

    return all_items


def parse_bird_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
    """Parse Bird response to match xai_x output format.

    Args:
        response: Raw Bird JSON response
        query: Original search query for relevance scoring

    Returns:
        List of normalized item dicts matching xai_x.parse_x_response() format.
    """
    items = []

    # Check for errors
    if "error" in response and response["error"]:
        _log(f"Bird error: {response['error']}")
        return items

    # Bird returns a list of tweets directly or under a key
    raw_items = response if isinstance(response, list) else response.get("items", response.get("tweets", []))

    if not isinstance(raw_items, list):
        return items

    for i, tweet in enumerate(raw_items):
        if not isinstance(tweet, dict):
            continue

        # Extract URL - Bird uses permanent_url or we construct from id
        url = tweet.get("permanent_url") or tweet.get("url", "")
        if not url and tweet.get("id"):
            # Try different field structures Bird might use
            author = tweet.get("author", {}) or tweet.get("user", {})
            screen_name = author.get("username") or author.get("screen_name", "")
            if screen_name:
                url = f"https://x.com/{screen_name}/status/{tweet['id']}"

        if not url:
            continue

        # Parse date from created_at/createdAt (e.g., "Wed Jan 15 14:30:00 +0000 2026")
        date = None
        created_at = tweet.get("createdAt") or tweet.get("created_at", "")
        if created_at:
            try:
                # Try ISO format first (e.g., "2026-02-03T22:33:32Z")
                # Check for ISO date separator, not just "T" (which appears in "Tue")
                if len(created_at) > 10 and created_at[10] == "T":
                    dt = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
                else:
                    # Twitter format: "Wed Jan 15 14:30:00 +0000 2026"
                    dt = datetime.strptime(created_at, "%a %b %d %H:%M:%S %z %Y")
                date = dt.strftime("%Y-%m-%d")
            except (ValueError, TypeError):
                pass

        # Extract user info (Bird uses author.username, older format uses user.screen_name)
        author = tweet.get("author", {}) or tweet.get("user", {})
        author_handle = author.get("username") or author.get("screen_name", "") or tweet.get("author_handle", "")

        # Build engagement dict (Bird uses camelCase: likeCount, retweetCount, etc.)
        engagement = {
            "likes": _first_of(tweet.get("likeCount"), tweet.get("like_count"), tweet.get("favorite_count")),
            "reposts": _first_of(tweet.get("retweetCount"), tweet.get("retweet_count")),
            "replies": _first_of(tweet.get("replyCount"), tweet.get("reply_count")),
            "quotes": _first_of(tweet.get("quoteCount"), tweet.get("quote_count")),
        }
        # Convert to int where possible
        for key in engagement:
            if engagement[key] is not None:
                try:
                    engagement[key] = int(engagement[key])
                except (ValueError, TypeError):
                    engagement[key] = None

        # Build normalized item
        item = {
            "id": f"X{i+1}",
            "text": str(tweet.get("text", tweet.get("full_text", ""))).strip()[:500],
            "url": url,
            "author_handle": author_handle.lstrip("@"),
            "date": date,
            "engagement": engagement,
            "why_relevant": "",  # Bird doesn't provide relevance explanations
            "relevance": _compute_relevance(query, str(tweet.get("text", ""))) if query else 0.7,
        }

        items.append(item)

    return items


path: scripts/lib/bluesky.py
content: """Bluesky search via AT Protocol (requires app password).

Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
"""

import math
import re
import sys
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional

from . import http, log

BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"

DEPTH_CONFIG = {
    "quick": 15,
    "default": 30,
    "deep": 60,
}

# Module-level token cache (valid for the lifetime of a single research run)
_cached_token: Optional[str] = None
_session_error: Optional[str] = None


def _log(msg: str):
    log.source_log("Bluesky", msg)


def _create_session(handle: str, app_password: str) -> Optional[str]:
    """Create an AT Protocol session and return the access token.

    Args:
        handle: Bluesky handle (e.g. user.bsky.social)
        app_password: App password from bsky.app/settings/app-passwords

    Returns:
        Access JWT string, or None on failure. Sets _session_error on failure.
    """
    global _cached_token, _session_error
    if _cached_token:
        return _cached_token

    try:
        response = http.request(
            "POST",
            BSKY_SESSION_URL,
            json_data={"identifier": handle, "password": app_password},
            timeout=15,
        )
        [REDACTED_SECRET]"accessJwt")
        if [REDACTED_SECRET] = token
            _session_error = None
            _log("Session created successfully")
            return token
        _log("No accessJwt in session response")
        _session_error = "No accessJwt in session response"
        return None
    except http.HTTPError as e:
        if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
            _session_error = "Cloudflare blocked the request (403 Forbidden). This is a network-level block, not an auth issue. Try a different network or VPN."
        elif e.status_code == 401:
            _session_error = "Invalid credentials (401 Unauthorized). Check BSKY_HANDLE and BSKY_APP_PASSWORD."
        else:
            _session_error = f"Session request failed: {e}"
        _log(f"Session creation failed: {_session_error}")
        return None
    except Exception as e:
        _session_error = f"Session request failed: {type(e).__name__}: {e}"
        _log(f"Session creation failed: {_session_error}")
        return None


def _extract_core_subject(topic: str) -> str:
    """Extract core subject from verbose query for Bluesky search."""
    from .query import extract_core_subject
    _BSKY_NOISE = frozenset({
        'best', 'top', 'good', 'great', 'awesome',
        'latest', 'new', 'news', 'update', 'updates',
        'trending', 'hottest', 'popular', 'viral',
        'practices', 'features', 'recommendations', 'advice',
    })
    return extract_core_subject(topic, noise=_BSKY_NOISE)


def _parse_date(item: Dict[str, Any]) -> Optional[str]:
    """Parse date from Bluesky post to YYYY-MM-DD.

    AT Protocol uses ISO 8601 format in indexedAt and createdAt fields.
    """
    for key in ("indexedAt", "createdAt"):
        val = item.get(key)
        if val and isinstance(val, str):
            try:
                dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
                return dt.strftime("%Y-%m-%d")
            except (ValueError, TypeError):
                pass
    return None


def search_bluesky(
    topic: str,
    from_date: str,
    to_date: str,
    depth: str = "default",
    config: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
    """Search Bluesky via AT Protocol API.

    Args:
        topic: Search topic
        from_date: Start date (YYYY-MM-DD)
        to_date: End date (YYYY-MM-DD)
        depth: 'quick', 'default', or 'deep'
        config: Config dict with BSKY_HANDLE and BSKY_APP_PASSWORD

    Returns:
        Dict with 'posts' list from AT Protocol response.
    """
    config = config or {}
    handle = config.get("BSKY_HANDLE", "")
    app_password = config.get("BSKY_APP_PASSWORD", "")

    if not handle or not app_password:
        return {"posts": [], "error": "Bluesky credentials not configured"}

    # Authenticate
    [REDACTED_SECRET], app_password)
    if not [REDACTED_SECRET] = _session_error or "Bluesky session creation failed (unknown error)"
        return {"posts": [], "error": error_msg}

    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)

    _log(f"Searching for '{core_topic}' (depth={depth}, limit={count})")

    from urllib.parse import urlencode
    params = {
        "q": core_topic,
        "limit": str(min(count, 100)),
        "sort": "top",
    }
    url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"

    try:
        response = http.request(
            "GET", url,
            headers={"Authorization": f"Bearer {token}"},
            timeout=30,
        )
    except http.HTTPError as e:
        _log(f"Search failed: {e}")
        if e.status_code == 403 and e.body and "cloudflare" in e.body.lower():
            return {"posts": [], "error": "Bluesky search blocked by Cloudflare (403). This is a network-level block - try a different network or VPN."}
        return {"posts": [], "error": f"Bluesky search failed: {e}"}
    except Exception as e:
        _log(f"Search failed: {e}")
        return {"posts": [], "error": f"Bluesky search failed: {type(e).__name__}: {e}"}

    posts = response.get("posts", [])
    _log(f"Found {len(posts)} posts")
    return response


def parse_bluesky_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
    """Parse AT Protocol response into normalized item dicts.

    Returns:
        List of item dicts ready for normalization.
    """
    posts = response.get("posts", [])
    items = []

    for i, post in enumerate(posts):
        record = post.get("record") or {}
        text = record.get("text") or ""

        author = post.get("author") or {}
        handle = author.get("handle") or ""
        display_name = author.get("displayName") or handle

        # Post URI -> URL
        # URI format: at://did:plc:xxx/app.bsky.feed.post/rkey
        uri = post.get("uri") or ""
        rkey = uri.rsplit("/", 1)[-1] if uri else ""
        url = f"https://bsky.app/profile/{handle}/post/{rkey}" if handle and rkey else ""

        likes = post.get("likeCount") or 0
        reposts = post.get("repostCount") or 0
        replies = post.get("replyCount") or 0
        quotes = post.get("quoteCount") or 0

        date_str = _parse_date(post) or _parse_date(record)

        # Relevance: position-based (AT Protocol sorts by relevance with sort=top)
        rank_score = max(0.3, 1.0 - (i * 0.02))
        engagement_boost = min(0.2, math.log1p(likes + reposts) / 40)
        relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)

        items.append({
            "handle": handle,
            "display_name": display_name,
            "text": text,
            "url": url,
            "date": date_str,
            "engagement": {
                "likes": likes,
                "reposts": reposts,
                "replies": replies,
                "quotes": quotes,
            },
            "relevance": round(relevance, 2),
            "why_relevant": f"Bluesky: @{handle}: {text[:60]}" if text else f"Bluesky: {handle}",
        })

    return items


path: scripts/lib/cluster.py
content: """Candidate clustering and representative selection."""

from __future__ import annotations

import re

from . import dedupe, schema

CLUSTERABLE_INTENTS = {"breaking_news", "opinion", "comparison", "prediction"}

# Words too common to signal shared topic between clusters.
_ENTITY_STOPWORDS = frozenset({
    "the", "a", "an", "to", "for", "how", "is", "in", "of", "on", "and",
    "with", "from", "by", "at", "this", "that", "it", "what", "are", "do",
    "can", "his", "her", "he", "she", "its", "was", "has", "new", "just",
    "says", "said", "will", "about", "after", "now", "all", "been", "here",
    "not", "out", "up", "more", "also", "but", "who", "year", "first",
    "make", "being", "making", "over", "into", "than", "they", "their",
    "would", "could", "get", "got", "some", "like", "back", "going",
    "breaking", "https", "http", "www", "com",
})


def _candidate_text(candidate: schema.Candidate) -> str:
    return " ".join(part for part in [candidate.title, candidate.snippet] if part).strip()


def _extract_entities(text: str) -> set[str]:
    """Extract significant words (proper nouns, numbers, capitalized words) from text.

    Used for cross-source cluster merging where phrasing differs but entities overlap.
    """
    # Normalize but preserve word boundaries
    words = re.sub(r"[^\w\s]", " ", text).split()
    entities = set()
    for word in words:
        lower = word.lower()
        if lower in _ENTITY_STOPWORDS or len(word) <= 2:
            continue
        # Keep words that are: capitalized, ALL CAPS, contain digits, or 4+ chars
        if word[0].isupper() or word.isupper() or any(c.isdigit() for c in word) or len(word) >= 4:
            entities.add(lower)
    return entities


def _entity_overlap(entities_a: set[str], entities_b: set[str]) -> float:
    """Jaccard-style overlap on extracted entities."""
    if not entities_a or not entities_b:
        return 0.0
    intersection = entities_a & entities_b
    smaller = min(len(entities_a), len(entities_b))
    # Use overlap coefficient (intersection / min) instead of Jaccard,
    # because a short tweet about the same event as a long Reddit post
    # will have fewer total entities but high overlap with the larger set.
    return len(intersection) / smaller if smaller > 0 else 0.0


def _mmr_representatives(
    candidates: list[schema.Candidate],
    limit: int = 3,
    diversity_lambda: float = 0.75,
) -> list[str]:
    selected: list[schema.Candidate] = []
    remaining = list(candidates)
    while remaining and len(selected) < limit:
        if not selected:
            best = max(remaining, key=lambda candidate: candidate.final_score)
            selected.append(best)
            remaining.remove(best)
            continue

        def score(candidate: schema.Candidate) -> float:
            diversity_penalty = max(
                dedupe.hybrid_similarity(_candidate_text(candidate), _candidate_text(existing))
                for existing in selected
            )
            return (diversity_lambda * candidate.final_score) - ((1 - diversity_lambda) * diversity_penalty * 100)

        best = max(remaining, key=score)
        selected.append(best)
        remaining.remove(best)
    return [candidate.candidate_id for candidate in selected]


def cluster_candidates(
    candidates: list[schema.Candidate],
    plan: schema.QueryPlan,
) -> list[schema.Cluster]:
    """Greedy clustering around high-ranked leaders."""
    if plan.intent not in CLUSTERABLE_INTENTS or plan.cluster_mode == "none":
        clusters = []
        for index, candidate in enumerate(candidates, start=1):
            cluster_id = f"cluster-{index}"
            candidate.cluster_id = cluster_id
            clusters.append(
                schema.Cluster(
                    cluster_id=cluster_id,
                    title=candidate.title,
                    candidate_ids=[candidate.candidate_id],
                    representative_ids=[candidate.candidate_id],
                    sources=sorted(schema.candidate_sources(candidate)),
                    score=candidate.final_score,
                    uncertainty=None,
                )
            )
        return clusters

    groups: list[list[schema.Candidate]] = []
    # Lower threshold for breaking_news: related articles share fewer exact
    # words but cover the same event.
    threshold = 0.42 if plan.intent == "breaking_news" else 0.48
    for candidate in candidates:
        assigned = False
        for group in groups:
            leader = group[0]
            similarity = dedupe.hybrid_similarity(_candidate_text(candidate), _candidate_text(leader))
            if similarity >= threshold:
                group.append(candidate)
                assigned = True
                break
        if not assigned:
            groups.append([candidate])

    clusters: list[schema.Cluster] = []
    for index, group in enumerate(groups, start=1):
        group.sort(key=lambda candidate: candidate.final_score, reverse=True)
        cluster_id = f"cluster-{index}"
        representatives = _mmr_representatives(group)
        for candidate in group:
            candidate.cluster_id = cluster_id
        clusters.append(
            schema.Cluster(
                cluster_id=cluster_id,
                title=group[0].title,
                candidate_ids=[candidate.candidate_id for candidate in group],
                representative_ids=representatives,
                sources=sorted({source for candidate in group for source in schema.candidate_sources(candidate)}),
                score=max(candidate.final_score for candidate in group),
                uncertainty=_cluster_uncertainty(group),
            )
        )

    # Second pass: merge small clusters that share entities across sources.
    clusters = _merge_entity_clusters(clusters, candidates)

    return sorted(clusters, key=lambda cluster: cluster.score, reverse=True)


def _merge_entity_clusters(
    clusters: list[schema.Cluster],
    all_candidates: list[schema.Candidate],
) -> list[schema.Cluster]:
    """Merge small clusters that cover the same story across different sources.

    The initial greedy pass uses text similarity which misses cross-source
    matches where phrasing differs. This second pass looks at entity overlap
    (proper nouns, names, numbers) to catch cases like:
      - Reddit: "Kanye West to headline all three nights of Wireless Festival 2026"
      - X: "BREAKING: Kanye West (Ye) is making his massive UK comeback!"
    """
    if len(clusters) < 2:
        return clusters

    candidate_map = {c.candidate_id: c for c in all_candidates}

    # Build entity sets per cluster
    cluster_entities: list[set[str]] = []
    for cl in clusters:
        entities: set[str] = set()
        for cid in cl.candidate_ids:
            cand = candidate_map.get(cid)
            if cand:
                entities |= _extract_entities(_candidate_text(cand))
        cluster_entities.append(entities)

    # Only merge clusters with <= 3 items (don't merge already-large clusters)
    merged_into: dict[int, int] = {}  # index -> merge target index
    for i in range(len(clusters)):
        if i in merged_into or len(clusters[i].candidate_ids) > 3:
            continue
        for j in range(i + 1, len(clusters)):
            if j in merged_into or len(clusters[j].candidate_ids) > 3:
                continue
            # Require different sources to merge (same-source should already be grouped)
            sources_i = set(clusters[i].sources)
            sources_j = set(clusters[j].sources)
            if sources_i == sources_j and len(sources_i) == 1:
                continue
            # Prevent Polymarket clusters from merging with non-Polymarket
            # clusters. Prediction markets about "Sam Altman equity" should not
            # merge into a news cluster about "Sam Altman rivalry" just because
            # both mention the same entity.
            poly_i = "polymarket" in sources_i
            poly_j = "polymarket" in sources_j
            if poly_i != poly_j:
                continue

            overlap = _entity_overlap(cluster_entities[i], cluster_entities[j])
            if overlap >= 0.45:
                merged_into[j] = i

    if not merged_into:
        return clusters

    # Build merged cluster list
    result: list[schema.Cluster] = []
    for i, cl in enumerate(clusters):
        if i in merged_into:
            continue
        # Collect all clusters merged into this one
        merge_sources = [i] + [j for j, target in merged_into.items() if target == i]
        if len(merge_sources) == 1:
            result.append(cl)
            continue

        # Combine candidates from all merged clusters
        combined_cids: list[str] = []
        combined_sources: set[str] = set()
        best_score = 0.0
        for idx in merge_sources:
            combined_cids.extend(clusters[idx].candidate_ids)
            combined_sources.update(clusters[idx].sources)
            best_score = max(best_score, clusters[idx].score)

        # Pick representatives from combined pool
        combined_candidates = [candidate_map[cid] for cid in combined_cids if cid in candidate_map]
        combined_candidates.sort(key=lambda c: c.final_score, reverse=True)
        reps = _mmr_representatives(combined_candidates)

        cluster_id = cl.cluster_id
        for cid in combined_cids:
            cand = candidate_map.get(cid)
            if cand:
                cand.cluster_id = cluster_id

        result.append(schema.Cluster(
            cluster_id=cluster_id,
            title=combined_candidates[0].title if combined_candidates else cl.title,
            candidate_ids=combined_cids,
            representative_ids=reps,
            sources=sorted(combined_sources),
            score=best_score,
            uncertainty=_cluster_uncertainty(combined_candidates),
        ))

    return result


def _cluster_uncertainty(group: list[schema.Candidate]) -> str | None:
    sources = {source for candidate in group for source in schema.candidate_sources(candidate)}
    if len(sources) == 1:
        return "single-source"
    if max(candidate.final_score for candidate in group) < 55:
        return "thin-evidence"
    return None


path: scripts/lib/dates.py
content: """Date utilities for last30days skill."""

from datetime import datetime, timedelta, timezone
from typing import Optional, Tuple


def get_date_range(days: int = 30) -> Tuple[str, str]:
    """Get the date range for the last N days.

    Returns:
        Tuple of (from_date, to_date) as YYYY-MM-DD strings
    """
    today = datetime.now(timezone.utc).date()
    from_date = today - timedelta(days=days)
    return from_date.isoformat(), today.isoformat()


def parse_date(date_str: Optional[str]) -> Optional[datetime]:
    """Parse a date string in various formats.

    Supports: YYYY-MM-DD, ISO 8601, Unix timestamp
    """
    if not date_str:
        return None

    # Try Unix timestamp (from Reddit)
    try:
        ts = float(date_str)
        return datetime.fromtimestamp(ts, tz=timezone.utc)
    except (ValueError, TypeError):
        pass

    # Try ISO formats
    formats = [
        "%Y-%m-%d",
        "%Y-%m-%dT%H:%M:%S",
        "%Y-%m-%dT%H:%M:%SZ",
        "%Y-%m-%dT%H:%M:%S%z",
        "%Y-%m-%dT%H:%M:%S.%f%z",
    ]

    for fmt in formats:
        try:
            dt = datetime.strptime(date_str, fmt)
            if dt.tzinfo is not None:
                return dt.astimezone(timezone.utc)
            return dt.replace(tzinfo=timezone.utc)
        except ValueError:
            continue

    return None


def timestamp_to_date(ts: Optional[float]) -> Optional[str]:
    """Convert Unix timestamp to YYYY-MM-DD string."""
    if ts is None:
        return None
    try:
        dt = datetime.fromtimestamp(ts, tz=timezone.utc)
        return dt.date().isoformat()
    except (ValueError, TypeError, OSError):
        return None


def get_date_confidence(date_str: Optional[str], from_date: str, to_date: str) -> str:
    """Determine confidence level for a date.

    Args:
        date_str: The date to check (YYYY-MM-DD or None)
        from_date: Start of valid range (YYYY-MM-DD)
        to_date: End of valid range (YYYY-MM-DD)

    Returns:
        'high', 'med', or 'low'
    """
    if not date_str:
        return 'low'

    try:
        dt = datetime.strptime(date_str, "%Y-%m-%d").date()
        start = datetime.strptime(from_date, "%Y-%m-%d").date()
        end = datetime.strptime(to_date, "%Y-%m-%d").date()

        return 'high' if start <= dt <= end else 'low'
    except ValueError:
        return 'low'


def days_ago(date_str: Optional[str]) -> Optional[int]:
    """Calculate how many days ago a date is.

    Returns None if date is invalid or missing.
    """
    if not date_str:
        return None

    try:
        dt = datetime.strptime(date_str, "%Y-%m-%d").date()
        today = datetime.now(timezone.utc).date()
        delta = today - dt
        return delta.days
    except ValueError:
        return None


def recency_score(date_str: Optional[str], max_days: int = 30) -> int:
    """Calculate recency score (0-100).

    0 days ago = 100, max_days ago = 0, clamped.
    """
    age = days_ago(date_str)
    if age is None:
        return 0  # Unknown date gets worst score

    if age < 0:
        return 100  # Future date (treat as today)
    if age >= max_days:
        return 0

    return int(100 * (1 - age / max_days))


path: scripts/lib/dedupe.py
content: """Within-source near-duplicate detection."""

from __future__ import annotations

import re

from . import schema

STOPWORDS = frozenset(
    {
        "the",
        "a",
        "an",
        "to",
        "for",
        "how",
        "is",
        "in",
        "of",
        "on",
        "and",
        "with",
        "from",
        "by",
        "at",
        "this",
        "that",
        "it",
        "what",
        "are",
        "do",
        "can",
    }
)


def normalize_text(text: str) -> str:
    text = re.sub(r"[^\w\s]", " ", text.lower())
    return re.sub(r"\s+", " ", text).strip()


def get_ngrams(text: str, n: int = 3) -> set[str]:
    text = normalize_text(text)
    if len(text) < n:
        return {text} if text else set()
    return {text[index:index + n] for index in range(len(text) - n + 1)}


def jaccard_similarity(left: set[str], right: set[str]) -> float:
    if not left or not right:
        return 0.0
    union = left | right
    if not union:
        return 0.0
    return len(left & right) / len(union)


def token_jaccard(text_a: str, text_b: str) -> float:
    tokens_a = {
        token
        for token in normalize_text(text_a).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    tokens_b = {
        token
        for token in normalize_text(text_b).split()
        if len(token) > 1 and token not in STOPWORDS
    }
    return jaccard_similarity(tokens_a, tokens_b)


def hybrid_similarity(text_a: str, text_b: str) -> float:
    return max(
        jaccard_similarity(get_ngrams(text_a), get_ngrams(text_b)),
        token_jaccard(text_a, text_b),
    )


def item_text(item: schema.SourceItem) -> str:
    parts = [item.title, item.body, item.author or "", item.container or ""]
    return " ".join(part for part in parts if part).strip()


def dedupe_items(items: list[schema.SourceItem], threshold: float = 0.7) -> list[schema.SourceItem]:
    """Remove near-duplicates while keeping earlier, better-scored items."""
    kept: list[schema.SourceItem] = []
    for item in items:
        text = item_text(item)
        if not text:
            kept.append(item)
            continue
        is_duplicate = False
        for existing in kept:
            if hybrid_similarity(text, item_text(existing)) >= threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            kept.append(item)
    return kept


path: scripts/lib/entity_extract.py
content: """Entity extraction from initial search results for supplemental searches."""

import re
from collections import Counter
from typing import Any, Dict, List

# Handles that appear too frequently to be useful for targeted search.
# These are generic/platform accounts, not topic-specific voices.
GENERIC_HANDLES = {
    "elonmusk", "openai", "google", "microsoft", "apple", "meta",
    "github", "youtube", "x", "twitter", "reddit", "wikipedia",
    "nytimes", "washingtonpost", "cnn", "bbc", "reuters",
    "verified", "jack", "sundarpichai",
}


def extract_entities(
    reddit_items: List[Dict[str, Any]],
    x_items: List[Dict[str, Any]],
    max_handles: int = 5,
    max_hashtags: int = 3,
    max_subreddits: int = 5,
) -> Dict[str, List[str]]:
    """Extract key entities from Phase 1 results for supplemental searches.

    Parses X results for @handles and #hashtags, Reddit results for subreddit
    names and cross-referenced communities.

    Args:
        reddit_items: Raw Reddit item dicts from Phase 1
        x_items: Raw X item dicts from Phase 1
        max_handles: Maximum handles to return
        max_hashtags: Maximum hashtags to return
        max_subreddits: Maximum subreddits to return

    Returns:
        Dict with keys: x_handles, x_hashtags, reddit_subreddits
    """
    handles = _extract_x_handles(x_items)
    hashtags = _extract_x_hashtags(x_items)
    subreddits = _extract_subreddits(reddit_items)

    return {
        "x_handles": handles[:max_handles],
        "x_hashtags": hashtags[:max_hashtags],
        "reddit_subreddits": subreddits[:max_subreddits],
    }


def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank @handles from X results.

    Sources handles from:
    1. author_handle field (who posted)
    2. @mentions in post text (who they're talking about/to)

    Returns handles ranked by frequency, filtered for generic accounts.
    """
    handle_counts = Counter()

    for item in x_items:
        # Author handle
        author = item.get("author_handle", "").strip().lstrip("@").lower()
        if author and author not in GENERIC_HANDLES:
            handle_counts[author] += 1

        # @mentions in text
        text = item.get("text", "")
        mentions = re.findall(r'@(\w{1,15})', text)
        for mention in mentions:
            mention_lower = mention.lower()
            if mention_lower not in GENERIC_HANDLES:
                handle_counts[mention_lower] += 1

    # Return all handles ranked by frequency
    return [h for h, _ in handle_counts.most_common()]


def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank #hashtags from X results.

    Returns hashtags ranked by frequency.
    """
    hashtag_counts = Counter()

    for item in x_items:
        text = item.get("text", "")
        tags = re.findall(r'#(\w{2,30})', text)
        for tag in tags:
            hashtag_counts[tag.lower()] += 1

    # Return all hashtags ranked by frequency
    return [f"#{t}" for t, _ in hashtag_counts.most_common()]


def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
    """Extract and rank subreddits from Reddit results.

    Sources from:
    1. subreddit field on each result
    2. Cross-references in comment text (e.g., "check out r/localLLaMA")

    Returns subreddits ranked by frequency.
    """
    sub_counts = Counter()

    for item in reddit_items:
        # Primary subreddit
        sub = item.get("subreddit", "").strip().lstrip("r/")
        if sub:
            sub_counts[sub] += 1

        # Cross-references in comment insights
        for insight in item.get("comment_insights", []):
            cross_refs = re.findall(r'r/(\w{2,30})', insight)
            for ref in cross_refs:
                sub_counts[ref] += 1

        # Cross-references in top comments
        for comment in item.get("top_comments", []):
            excerpt = comment.get("excerpt", "")
            cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
            for ref in cross_refs:
                sub_counts[ref] += 1

    # Return subreddits ranked by frequency
    return [sub for sub, _ in sub_counts.most_common()]


path: scripts/lib/env.py
content: """Environment and API key management for last30days skill."""

from __future__ import annotations

import base64
import binascii
import json
import os
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal

# Allow override via environment variable for testing
# Set LAST30DAYS_CONFIG_DIR="" for clean/no-config mode
# Set LAST30DAYS_CONFIG_DIR="/path/to/dir" for custom config location
_config_override = os.environ.get('LAST30DAYS_CONFIG_DIR')
if _config_override == "":
    # Empty string = no config file (clean mode)
    CONFIG_DIR = None
    CONFIG_FILE = None
elif _config_override:
    CONFIG_DIR = Path(_config_override)
    CONFIG_FILE = CONFIG_DIR / ".env"
else:
    CONFIG_DIR = Path.home() / ".config" / "last30days"
    CONFIG_FILE = CONFIG_DIR / ".env"

CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))

AuthSource = Literal["api_key", "codex", "none"]
AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]

AUTH_SOURCE_API_KEY: AuthSource = "api_key"
AUTH_SOURCE_CODEX: AuthSource = "codex"
AUTH_SOURCE_NONE: AuthSource = "none"

AUTH_STATUS_OK: AuthStatus = "ok"
AUTH_STATUS_MISSING: AuthStatus = "missing"
AUTH_STATUS_EXPIRED: AuthStatus = "expired"
AUTH_STATUS_MISSING_ACCOUNT_ID: AuthStatus = "missing_account_id"


@dataclass(frozen=True)
class OpenAIAuth:
    token: str | None
    source: AuthSource
    status: AuthStatus
    account_id: str | None
    codex_auth_file: str


def _check_file_permissions(path: Path) -> None:
    """Warn to stderr if a secrets file has overly permissive permissions."""
    try:
        mode = path.stat().st_mode
        # Check if group or other can read (bits 0o044)
        if mode & 0o044:
            sys.stderr.write(
                f"[last30days] WARNING: {path} is readable by other users. "
                f"Run: chmod 600 {path}\n"
            )
            sys.stderr.flush()
    except OSError as exc:
        sys.stderr.write(f"[last30days] WARNING: could not stat {path}: {exc}\n")
        sys.stderr.flush()


def load_env_file(path: Path) -> dict[str, str]:
    """Load environment variables from a file."""
    env = {}
    if not path or not path.exists():
        return env
    _check_file_permissions(path)

    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if not line or line.startswith('#'):
                continue
            if '=' in line:
                key, _, value = line.partition('=')
                key = key.strip()
                value = value.strip()
                # Remove quotes if present
                if value and value[0] in ('"', "'") and value[-1] == value[0]:
                    value = value[1:-1]
                if key and value:
                    env[key] = value
    return env


def _decode_jwt_payload(token: str) -> dict[str, Any] | None:
    """Decode JWT payload without verification."""
    try:
        parts = token.split(".")
        if len(parts) < 2:
            return None
        payload_b64 = parts[1]
        pad = "=" * (-len(payload_b64) % 4)
        decoded = base64.urlsafe_b64decode(payload_b64 + pad)
        return json.loads(decoded.decode("utf-8"))
    except (json.JSONDecodeError, UnicodeDecodeError, binascii.Error, IndexError) as exc:
        sys.stderr.write(f"[last30days] WARNING: malformed JWT [REDACTED_SECRET]")
        sys.stderr.flush()
        return None


def _token_expired(token: str, leeway_seconds: int = 60) -> bool:
    """Check if JWT token is expired."""
    payload = _decode_jwt_payload(token)
    if not payload:
        return False
    exp = payload.get("exp")
    if not exp:
        return False
    return exp <= (time.time() + leeway_seconds)


def extract_chatgpt_account_id(access_token: str) -> str | None:
    """Extract chatgpt_account_id from JWT token."""
    payload = _decode_jwt_payload(access_token)
    if not payload:
        return None
    auth_claim = payload.get("https://api.openai.com/auth", {})
    if isinstance(auth_claim, dict):
        return auth_claim.get("chatgpt_account_id")
    return None


def load_codex_auth(path: Path = CODEX_AUTH_FILE) -> dict[str, Any]:
    """Load Codex auth JSON."""
    if not path.exists():
        return {}
    try:
        with open(path, "r") as f:
            return json.load(f)
    except json.JSONDecodeError:
        sys.stderr.write(
            f"[last30days] WARNING: {path} exists but contains invalid JSON -- ignoring\n"
        )
        sys.stderr.flush()
        return {}


def get_codex_access_token() -> tuple[str | None, str]:
    """Get Codex access token from auth.json.

    Returns:
        (token, status) where status is 'ok', 'missing', or 'expired'
    """
    auth = load_codex_auth()
    token = None
    if isinstance(auth, dict):
        tokens = auth.get("tokens") or {}
        if isinstance(tokens, dict):
            [REDACTED_SECRET]"access_token")
        if not [REDACTED_SECRET]"access_token")
    if not [REDACTED_SECRET] None, AUTH_STATUS_MISSING
    if _token_expired(token):
        return None, AUTH_STATUS_EXPIRED
    return token, AUTH_STATUS_OK


def get_openai_auth(file_env: dict[str, str]) -> OpenAIAuth:
    """Resolve OpenAI auth from API key or Codex login."""
    [REDACTED_SECRET]'OPENAI_API_KEY') or file_env.get('OPENAI_API_KEY')
    if [REDACTED_SECRET] OpenAIAuth(
            [REDACTED_SECRET],
            source=AUTH_SOURCE_API_KEY,
            status=AUTH_STATUS_OK,
            account_id=None,
            codex_auth_file=str(CODEX_AUTH_FILE),
        )

    # Codex auth (chatgpt.com backend) intentionally skipped.
    # The endpoint is unstable and causes crashes when the token expires.
    # Users who want OpenAI should set OPENAI_API_KEY explicitly.

    return OpenAIAuth(
        token=None,
        source=AUTH_SOURCE_NONE,
        status=AUTH_STATUS_MISSING,
        account_id=None,
        codex_auth_file=str(CODEX_AUTH_FILE),
    )


def _find_project_env() -> Path | None:
    """Find per-project .env by walking up from cwd.

    Searches for .claude/last30days.env in each parent directory,
    stopping at the user's home directory or filesystem root.
    """
    cwd = Path.cwd()
    for parent in [cwd, *cwd.parents]:
        candidate = parent / '.claude' / 'last30days.env'
        if candidate.exists():
            return candidate
        # Stop at filesystem root or home
        if parent == Path.home() or parent == parent.parent:
            break
    return None


def get_config() -> dict[str, Any]:
    """Load configuration from multiple sources.

    Priority (highest wins):
      1. Environment variables (os.environ)
      2. .claude/last30days.env (per-project config)
      3. ~/.config/last30days/.env (global config)
    """
    # Load from global config file
    file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}

    # Load from per-project config (overrides global)
    project_env_path = _find_project_env()
    project_env = load_env_file(project_env_path) if project_env_path else {}

    # Merge: project overrides global
    merged_env = {**file_env, **project_env}

    openai_auth = get_openai_auth(merged_env)

    # Build config: Codex/OpenAI auth + process.env > project .env > global .env
    config = {
        'OPENAI_API_KEY': openai_auth.token,
        'OPENAI_AUTH_SOURCE': openai_auth.source,
        'OPENAI_AUTH_STATUS': openai_auth.status,
        'OPENAI_CHATGPT_ACCOUNT_ID': openai_auth.account_id,
        'CODEX_AUTH_FILE': openai_auth.codex_auth_file,
    }

    keys = [
        ('XAI_API_KEY', None),
        ('GOOGLE_API_KEY', None),
        ('GEMINI_API_KEY', None),
        ('GOOGLE_GENAI_API_KEY', None),
        ('XIAOHONGSHU_API_BASE', None),
        ('LAST30DAYS_REASONING_PROVIDER', 'auto'),
        ('LAST30DAYS_PLANNER_MODEL', None),
        ('LAST30DAYS_RERANK_MODEL', None),
        ('LAST30DAYS_X_MODEL', None),
        ('LAST30DAYS_X_BACKEND', None),
        ('OPENAI_MODEL_PIN', None),
        ('XAI_MODEL_PIN', None),
        ('SCRAPECREATORS_API_KEY', None),
        ('APIFY_API_TOKEN', None),
        ('AUTH_TOKEN', None),
        ('CT0', None),
        ('BSKY_HANDLE', None),
        ('BSKY_APP_PASSWORD', None),
        ('TRUTHSOCIAL_TOKEN', None),
        ('BRAVE_API_KEY', None),
        ('EXA_API_KEY', None),
        ('SERPER_API_KEY', None),
        ('OPENROUTER_API_KEY', None),
        ('PARALLEL_API_KEY', None),
        ('FROM_BROWSER', None),
        ('SETUP_COMPLETE', None),
        ('INCLUDE_SOURCES', None),
    ]

    for key, default in keys:
        config[key] = os.environ.get(key) or merged_env.get(key, default)

    # Track which config source was used
    if project_env_path:
        config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
    elif CONFIG_FILE and CONFIG_FILE.exists():
        config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
    else:
        config['_CONFIG_SOURCE'] = 'env_only'

    # Extract browser credentials if configured
        if not config.get(key):
            config[key] = value
            config[f"_{key}_SOURCE"] = "browser"

    return config




def get_x_source_with_method(config: dict[str, Any]) -> tuple[str | None, str]:
    """Return (source, method) for X search, where method describes the auth origin."""
    if config.get("XAI_API_KEY"):
        return "xai", "xai"
    if config.get("AUTH_TOKEN") and config.get("CT0"):
        method = config.get("_AUTH_TOKEN_SOURCE", "env")
        return "bird", method
    return None, "none"


def config_exists() -> bool:
    """Check if any configuration source exists."""
    if _find_project_env():
        return True
    if CONFIG_FILE:
        return CONFIG_FILE.exists()
    return False


def is_reddit_available(config: dict[str, Any]) -> bool:
    """Check if Reddit search is available.

    v3 uses ScrapeCreators only.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))


def get_reddit_source(config: dict[str, Any]) -> str | None:
    """Determine which Reddit backend to use.

    Returns: 'scrapecreators' or None
    """
    if config.get('SCRAPECREATORS_API_KEY'):
        return 'scrapecreators'
    return None


def get_x_source(config: dict[str, Any]) -> str | None:
    """Determine the best available explicit X/Twitter source.

    Priority: explicit backend pin, then xAI, then Bird with explicit credentials.

        access causes popups during normal pipeline runs. Bird is only considered
    available when AUTH_TOKEN and CT0 are present explicitly.

    Args:
        config: Configuration dict from get_config()

    Returns:
        'bird' if Bird is installed and explicit cookies are configured,
        'xai' if XAI_API_KEY is configured,
        None if no X source available.
    """
    # Import here to avoid circular dependency
    from . import bird_x

    preferred = (config.get('LAST30DAYS_X_BACKEND') or '').lower()
    has_bird_creds = bool(config.get('AUTH_TOKEN') and config.get('CT0'))
    if has_bird_creds:
        bird_x.set_credentials(config.get('AUTH_TOKEN'), config.get('CT0'))

    if preferred == 'xai':
        return 'xai' if config.get('XAI_API_KEY') else None
    if preferred == 'bird':
        return 'bird' if has_bird_creds and bird_x.is_bird_installed() else None

    if config.get('XAI_API_KEY'):
        return 'xai'
    if has_bird_creds and bird_x.is_bird_installed():
        return 'bird'

    return None


def is_ytdlp_available() -> bool:
    """Check if yt-dlp is installed for YouTube search."""
    from . import youtube_yt
    return youtube_yt.is_ytdlp_installed()


def is_youtube_comments_available(config: dict[str, Any]) -> bool:
    """Check if YouTube comment enrichment is available.

    Requires SCRAPECREATORS_API_KEY AND youtube_comments in INCLUDE_SOURCES.
    """
    if not config.get('SCRAPECREATORS_API_KEY'):
        return False
    include = _parse_include_sources(config)
    return 'youtube_comments' in include


def is_youtube_sc_available(config: dict[str, Any]) -> bool:
    """Check if ScrapeCreators YouTube search fallback is available.

    Used when yt-dlp is not installed or fails.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))


def is_hackernews_available() -> bool:
    """Check if Hacker News source is available.

    Always returns True - HN uses free Algolia API, no key needed.
    """
    return True


def is_bluesky_available(config: dict[str, Any]) -> bool:
    """Check if Bluesky source is available.

    Requires BSKY_HANDLE and BSKY_APP_PASSWORD (app password from bsky.app/settings).
    """
    return bool(config.get('BSKY_HANDLE') and config.get('BSKY_APP_PASSWORD'))


def is_truthsocial_available(config: dict[str, Any]) -> bool:
    """Check if Truth Social source is available.

    Requires TRUTHSOCIAL_TOKEN (bearer token from browser dev tools).
    """
    return bool(config.get('TRUTHSOCIAL_TOKEN'))


def is_polymarket_available() -> bool:
    """Check if Polymarket source is available.

    Always returns True - Gamma API is free, no key needed.
    """
    return True


def is_tiktok_available(config: dict[str, Any]) -> bool:
    """Check if TikTok source is available (ScrapeCreators or legacy Apify).

    Returns True if SCRAPECREATORS_API_KEY or APIFY_API_TOKEN is set.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN'))


def get_tiktok_token(config: dict[str, Any]) -> str:
    """Get TikTok API token, preferring ScrapeCreators over legacy Apify."""
    return config.get('SCRAPECREATORS_API_KEY') or config.get('APIFY_API_TOKEN') or ''


def _parse_include_sources(config: dict[str, Any]) -> set[str]:
    """Parse INCLUDE_SOURCES config value into a set of lowercase source names."""
    raw = config.get('INCLUDE_SOURCES') or ''
    return {s.strip().lower() for s in raw.split(',') if s.strip()}


def is_threads_available(config: dict[str, Any]) -> bool:
    """Check if Threads source is available.

    Requires SCRAPECREATORS_API_KEY AND 'threads' in INCLUDE_SOURCES.
    Threads is an opt-in source - it is not activated by default.
    """
    if not config.get('SCRAPECREATORS_API_KEY'):
        return False
    return 'threads' in _parse_include_sources(config)


def is_instagram_available(config: dict[str, Any]) -> bool:
    """Check if Instagram source is available (ScrapeCreators).

    Returns True if SCRAPECREATORS_API_KEY is set.
    Instagram uses the same key as TikTok.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))


def get_instagram_token(config: dict[str, Any]) -> str:
    """Get Instagram API token (same ScrapeCreators key as TikTok)."""
    return config.get('SCRAPECREATORS_API_KEY') or ''


def get_xiaohongshu_api_base(config: dict[str, Any]) -> str:
    """Get Xiaohongshu HTTP API base URL.

    Defaults to host.docker.internal so OpenClaw Docker can reach host service.
    """
    return (config.get('XIAOHONGSHU_API_BASE') or "http://host.docker.internal:18060").rstrip("/")


def is_xiaohongshu_available(config: dict[str, Any]) -> bool:
    """Check whether Xiaohongshu HTTP API is reachable and logged in."""
    # Import here to avoid heavy imports at module load.
    from . import http

    base = get_xiaohongshu_api_base(config)
    try:
        # Keep health probe snappy, but allow one retry for transient hiccups.
        health = http.get(f"{base}/health", timeout=3, retries=2)
        if not isinstance(health, dict):
            return False
        if not health.get("success"):
            return False

        # Login probe can be slower on some deployments (browser/session checks),
        # so use a slightly longer timeout to avoid false negatives.
        login = http.get(f"{base}/api/v1/login/status", timeout=8, retries=2)
        is_logged_in = (
            login.get("data", {}).get("is_logged_in")
            if isinstance(login, dict) else False
        )
        return bool(is_logged_in)
    except (OSError, http.HTTPError):
        return False
    except Exception as exc:
        sys.stderr.write(
            f"[last30days] WARNING: unexpected error checking Xiaohongshu: "
            f"{type(exc).__name__}: {exc}\n"
        )
        sys.stderr.flush()
        return False


# Backward compat alias
is_apify_available = is_tiktok_available


def get_x_source_status(config: dict[str, Any]) -> dict[str, Any]:
    """Get detailed X source status for UI decisions.

    Returns:
        Dict with keys: source, bird_installed, bird_authenticated,
        bird_username, xai_available, can_install_bird
    """
    from . import bird_x

    bird_status = bird_x.get_bird_status()
    xai_available = bool(config.get('XAI_API_KEY'))

    # Determine active source
    if bird_status["authenticated"]:
        source = 'bird'
    elif xai_available:
        source = 'xai'
    else:
        source = None

    return {
        "source": source,
        "bird_installed": bird_status["installed"],
        "bird_authenticated": bird_status["authenticated"],
        "bird_username": bird_status["username"],
        "xai_available": xai_available,
        "can_install_bird": bird_status["can_install"],
    }


# Pinterest
def is_pinterest_available(config: dict[str, Any]) -> bool:
    """Check if Pinterest source is available.

    Returns True when SCRAPECREATORS_API_KEY is set AND 'pinterest' is in
    INCLUDE_SOURCES (or requested_sources at the pipeline level).  Pinterest
    is opt-in because not every topic benefits from visual pin results.
    """
    return bool(config.get('SCRAPECREATORS_API_KEY'))


def get_pinterest_token(config: dict[str, Any]) -> str:
    """Get Pinterest API token (same ScrapeCreators key as TikTok/Instagram)."""
    return config.get('SCRAPECREATORS_API_KEY') or ''


path: scripts/lib/fusion.py
content: """Weighted reciprocal rank fusion for per-(subquery, source) streams."""

from __future__ import annotations

from urllib.parse import parse_qs, urlencode, urlparse, urlunparse

from . import schema

# Standard RRF smoothing constant (Cormack et al. 2009)
RRF_K = 60


def _candidate_sort_key(c: schema.Candidate) -> tuple:
    return (-c.rrf_score, -c.local_relevance, -c.freshness, schema.candidate_source_label(c), c.title)


def _normalize_url(url: str) -> str:
    """Normalize URL for dedup: lowercase, strip www/old/m prefixes, remove tracking params."""
    parsed = urlparse(url.strip().lower())
    netloc = parsed.netloc
    for prefix in ("www.", "old.", "m."):
        if netloc.startswith(prefix):
            netloc = netloc[len(prefix):]
    # Strip tracking params
    params = parse_qs(parsed.query)
    clean_params = {k: v for k, v in params.items() if not k.startswith("utm_")}
    query = urlencode(clean_params, doseq=True)
    return urlunparse((parsed.scheme, netloc, parsed.path.rstrip("/"), "", query, ""))


def candidate_key(item: schema.SourceItem) -> str:
    if item.url:
        return _normalize_url(item.url)
    return f"{item.source}:{item.item_id}"


_DIVERSITY_RELEVANCE_THRESHOLD = 0.25

# Per-author cap: no single author/handle should dominate the pool.
_MAX_ITEMS_PER_AUTHOR = 3


def _extract_author(candidate: schema.Candidate) -> str | None:
    """Return a normalized author key from a candidate's source items."""
    for item in candidate.source_items:
        if item.author:
            return item.author.strip().lower()
    return None


def _apply_per_author_cap(
    candidates: list[schema.Candidate],
    max_per_author: int = _MAX_ITEMS_PER_AUTHOR,
) -> list[schema.Candidate]:
    """Keep at most *max_per_author* items from any single author.

    Candidates are assumed to already be sorted by quality (rrf_score etc.),
    so the first N encountered per author are the best ones.
    """
    author_counts: dict[str, int] = {}
    result: list[schema.Candidate] = []
    for c in candidates:
        author = _extract_author(c)
        if author is None:
            result.append(c)
            continue
        count = author_counts.get(author, 0)
        if count < max_per_author:
            result.append(c)
            author_counts[author] = count + 1
    return result


def _diversify_pool(
    fused: list[schema.Candidate],
    pool_limit: int,
    min_per_source: int = 2,
) -> list[schema.Candidate]:
    """Ensure at least *min_per_source* items per qualifying source survive truncation.

    Sources only qualify for reserved slots if their best item exceeds
    the relevance threshold. Low-relevance sources compete on merit only.
    """
    max_relevance: dict[str, float] = {}
    for c in fused:
        current = max_relevance.get(c.source, 0.0)
        if c.local_relevance > current:
            max_relevance[c.source] = c.local_relevance

    reserved: dict[str, list[schema.Candidate]] = {}
    remainder: list[schema.Candidate] = []
    for c in fused:
        qualifies = max_relevance.get(c.source, 0.0) >= _DIVERSITY_RELEVANCE_THRESHOLD
        bucket = reserved.setdefault(c.source, [])
        if qualifies and len(bucket) < min_per_source:
            bucket.append(c)
        else:
            remainder.append(c)
    pool = [c for per_source in reserved.values() for c in per_source]
    seen = {c.candidate_id for c in pool}
    for c in remainder:
        if len(pool) >= pool_limit:
            break
        if c.candidate_id not in seen:
            pool.append(c)
    pool.sort(key=_candidate_sort_key)
    return pool[:pool_limit]


def weighted_rrf(
    streams: dict[tuple[str, str], list[schema.SourceItem]],
    plan: schema.QueryPlan,
    *,
    pool_limit: int,
) -> list[schema.Candidate]:
    """Fuse ranked lists into a single candidate pool."""
    subqueries = {subquery.label: subquery for subquery in plan.subqueries}
    candidates: dict[str, schema.Candidate] = {}

    for (label, source), items in streams.items():
        subquery = subqueries[label]
        weight = subquery.weight * plan.source_weights.get(source, 1.0)
        for rank, item in enumerate(items, start=1):
            key = candidate_key(item)
            score = weight / (RRF_K + rank)
            item_local_relevance = item.local_relevance if item.local_relevance is not None else float(item.metadata.get("local_relevance", item.relevance_hint))
            item_freshness = item.freshness if item.freshness is not None else int(item.metadata.get("freshness", 0))
            item_source_quality = item.source_quality if item.source_quality is not None else float(item.metadata.get("source_quality", 0.6))
            if key not in candidates:
                candidates[key] = schema.Candidate(
                    candidate_id=key,
                    item_id=item.item_id,
                    source=item.source,
                    title=item.title,
                    url=item.url,
                    snippet=item.snippet,
                    subquery_labels=[label],
                    native_ranks={f"{label}:{source}": rank},
                    local_relevance=item_local_relevance,
                    freshness=item_freshness,
                    engagement=item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score"),
                    source_quality=item_source_quality,
                    rrf_score=score,
                    sources=[item.source],
                    source_items=[item],
                    metadata={
                        "provenance": [
                            {
                                "source": source,
                                "subquery_label": label,
                                "native_rank": rank,
                                "item_id": item.item_id,
                            }
                        ]
                    },
                )
                continue

            candidate = candidates[key]
            candidate.rrf_score += score
            previous_primary_score = (candidate.local_relevance * 100.0) + candidate.freshness + (candidate.source_quality * 10.0)
            incoming_primary_score = (item_local_relevance * 100.0) + item_freshness + (item_source_quality * 10.0)
            candidate.local_relevance = max(
                candidate.local_relevance,
                item_local_relevance,
            )
            candidate.freshness = max(candidate.freshness, item_freshness)
            item_eng = item.engagement_score if item.engagement_score is not None else item.metadata.get("engagement_score")
            if candidate.engagement is None:
                candidate.engagement = item_eng
            elif item_eng is not None:
                candidate.engagement = max(candidate.engagement, item_eng)
            candidate.source_quality = max(
                candidate.source_quality,
                item_source_quality,
            )
            candidate.native_ranks[f"{label}:{source}"] = rank
            if label not in candidate.subquery_labels:
                candidate.subquery_labels.append(label)
            if item.source not in candidate.sources:
                candidate.sources.append(item.source)
            if not any(existing.source == item.source and existing.item_id == item.item_id for existing in candidate.source_items):
                candidate.source_items.append(item)
            candidate.metadata.setdefault("provenance", []).append(
                {
                    "source": source,
                    "subquery_label": label,
                    "native_rank": rank,
                    "item_id": item.item_id,
                }
            )
            if incoming_primary_score > previous_primary_score:
                candidate.item_id = item.item_id
                candidate.source = item.source
                candidate.title = item.title
                candidate.snippet = item.snippet
            if len(candidate.snippet.split()) < len(item.snippet.split()):
                candidate.snippet = item.snippet

    fused = sorted(candidates.values(), key=_candidate_sort_key)
    fused = _apply_per_author_cap(fused)
    return _diversify_pool(fused, pool_limit)


path: scripts/lib/grounding.py
content: """Web search retrieval via Brave Search, Exa, and Serper."""

from __future__ import annotations

import urllib.parse
from datetime import datetime
from urllib.parse import urlparse

from . import dates, http


# ---------------------------------------------------------------------------
# Brave Search API
# ---------------------------------------------------------------------------

def brave_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
    url = (
        "https://api.search.brave.com/res/v1/web/search?"
        + urllib.parse.urlencode(
            {
                "q": query,
                "count": count,
                "freshness": f"{date_range[0]}to{date_range[1]}",
            }
        )
    )
    data = http.request("GET", url, headers={"X-Subscription-Token": api_key}, timeout=15)
    items = []
    for i, r in enumerate((data.get("web", {}).get("results", []))[:count]):
        raw_date = r.get("page_age") or ""
        pub_date = _normalize_date(raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WB{i + 1}",
            "title": r.get("title", ""),
            "url": r.get("url", ""),
            "source_domain": _domain(r.get("url", "")),
            "snippet": r.get("description", ""),
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Brave web search",
        })
    artifact = {"label": "brave", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact


# ---------------------------------------------------------------------------
# Exa AI Search
# ---------------------------------------------------------------------------

def exa_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://api.exa.ai/search",
        headers={"x-api-key": api_key},
        json_data={
            "query": query,
            "type": "auto",
            "numResults": count,
            "startPublishedDate": f"{date_range[0]}T00:00:00.000Z",
            "endPublishedDate": f"{date_range[1]}T23:59:59.999Z",
            "contents": {"text": {"maxCharacters": 2000}},
        },
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("results", []))[:count]):
        if not isinstance(r, dict):
            continue
        url = r.get("url", "")
        if not url:
            continue
        raw_date = r.get("publishedDate") or ""
        pub_date = _normalize_date(raw_date.split("T")[0] if "T" in raw_date else raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WE{i + 1}",
            "title": r.get("title", ""),
            "url": url,
            "source_domain": _domain(url),
            "snippet": (r.get("text") or "")[:500],
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Exa web search",
        })
    artifact = {"label": "exa", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact


# ---------------------------------------------------------------------------
# Serper (Google Search wrapper)
# ---------------------------------------------------------------------------

def serper_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://google.serper.dev/search",
        headers={"X-API-KEY": api_key},
        json_data={
            "q": query,
            "num": count,
            "tbs": f"cdr:1,cd_min:{_serper_date_param(date_range[0])},cd_max:{_serper_date_param(date_range[1])}",
        },
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("organic", []))[:count]):
        raw_date = r.get("date") or ""
        pub_date = _parse_serper_date(raw_date)
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WS{i + 1}",
            "title": r.get("title", ""),
            "url": r.get("link", ""),
            "source_domain": _domain(r.get("link", "")),
            "snippet": r.get("snippet", ""),
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Serper web search",
        })
    artifact = {"label": "serper", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact


# ---------------------------------------------------------------------------
# Parallel AI Search
# ---------------------------------------------------------------------------

def parallel_search(
    query: str, date_range: tuple[str, str], api_key: str, count: int = 5,
) -> tuple[list[dict], dict]:
    data = http.request(
        "POST", "https://api.parallel.ai/v1/search",
        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
        json_data={"query": query, "max_results": count},
        timeout=15,
    )
    items = []
    for i, r in enumerate((data.get("results", []))[:count]):
        if not isinstance(r, dict):
            continue
        url = r.get("url", "")
        if not url:
            continue
        raw_date = r.get("published_date") or ""
        pub_date = _normalize_date(raw_date[:10]) if raw_date else None
        if not _in_date_range(pub_date, date_range):
            continue
        items.append({
            "id": f"WP{i + 1}",
            "title": r.get("title", ""),
            "url": url,
            "source_domain": _domain(url),
            "snippet": r.get("snippet", ""),
            "date": pub_date,
            "relevance": 0.8,
            "why_relevant": "Parallel AI web search",
        })
    artifact = {"label": "parallel", "webSearchQueries": [query], "resultCount": len(items)}
    return items, artifact


def _parse_serper_date(raw: str) -> str | None:
    if not raw:
        return None
    normalized = _normalize_date(raw)
    if normalized:
        return normalized
    for fmt in ("%b %d, %Y", "%B %d, %Y", "%Y-%m-%d"):
        try:
            return datetime.strptime(raw.strip(), fmt).date().isoformat()
        except ValueError:
            continue
    return None




# ---------------------------------------------------------------------------
# Dispatcher
# ---------------------------------------------------------------------------

def web_search(
    query: str,
    date_range: tuple[str, str],
    config: dict,
    backend: str = "auto",
) -> tuple[list[dict], dict]:
    """Run web search with the specified or auto-detected backend."""
    if backend == "auto":
        if config.get("BRAVE_API_KEY"):
            backend = "brave"
        elif config.get("EXA_API_KEY"):
            backend = "exa"
        elif config.get("SERPER_API_KEY"):
            backend = "serper"
        elif config.get("PARALLEL_API_KEY"):
            backend = "parallel"
        else:
            return [], {}
    if backend == "brave":
        key = config.get("BRAVE_API_KEY")
        if not key:
            raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'")
        return brave_search(query, date_range, key)
    if backend == "exa":
        key = config.get("EXA_API_KEY")
        if not key:
            raise RuntimeError("EXA_API_KEY is required when web_backend='exa'")
        return exa_search(query, date_range, key)
    if backend == "serper":
        key = config.get("SERPER_API_KEY")
        if not key:
            raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'")
        return serper_search(query, date_range, key)
    if backend == "parallel":
        key = config.get("PARALLEL_API_KEY")
        if not key:
            raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'")
        return parallel_search(query, date_range, key)
    if backend != "none":
        raise ValueError(f"Unsupported web backend: {backend!r}")
    return [], {}


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def _normalize_date(value: object) -> str | None:
    if value is None:
        return None
    parsed = dates.parse_date(str(value).strip())
    if not parsed:
        return None
    return parsed.date().isoformat()


def _serper_date_param(iso_date: str) -> str:
    """Convert YYYY-MM-DD to MM/DD/YYYY for Serper tbs parameter."""
    parts = iso_date.split("-")
    return f"{parts[1]}/{parts[2]}/{parts[0]}"


def _in_date_range(pub_date: str | None, date_range: tuple[str, str]) -> bool:
    if not pub_date:
        return False
    return date_range[0] <= pub_date <= date_range[1]


def _domain(url: str) -> str:
    return urlparse(url).netloc.strip().lower()


path: scripts/lib/http.py
content: """HTTP utilities for last30days skill (stdlib only)."""

import json
import re
import sys
import time
import urllib.error
import urllib.request
from typing import Any, Dict, Optional, Union
from urllib.parse import urlencode

from . import log as _log

DEFAULT_TIMEOUT = 30


def log(msg: str):
    """Log debug message to stderr."""
    _log.debug(msg)


MAX_RETRIES = 5
MAX_429_RETRIES = 2
RETRY_DELAY = 2.0
USER_AGENT = "last30days-skill/3.0 (Assistant Skill)"


class HTTPError(Exception):
    """HTTP request error with status code."""
    def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
        super().__init__(message)
        self.status_code = status_code
        self.body = body


def request(
    method: str,
    url: str,
    headers: Optional[Dict[str, str]] = None,
    json_data: Optional[Dict[str, Any]] = None,
    timeout: int = DEFAULT_TIMEOUT,
    retries: int = MAX_RETRIES,
    max_429_retries: int = MAX_429_RETRIES,
    raw: bool = False,
) -> Union[Dict[str, Any], str]:
    """Make an HTTP request and return JSON response.

    Args:
        method: HTTP method (GET, POST, etc.)
        url: Request URL
        headers: Optional headers dict
        json_data: Optional JSON body (for POST)
        timeout: Request timeout in seconds
        retries: Number of retries on failure
        max_429_retries: Maximum 429 retries before giving up (separate cap)
        raw: If True, return raw response text instead of parsed JSON

    Returns:
        Parsed JSON response as dict, or raw text string if raw=True.

    Raises:
        HTTPError: On request failure
    """
    headers = headers or {}
    headers.setdefault("User-Agent", USER_AGENT)

    data = None
    if json_data is not None:
        data = json.dumps(json_data).encode('utf-8')
        headers.setdefault("Content-Type", "application/json")

    req = urllib.request.Request(url, data=data, headers=headers, method=method)

    safe_url = re.sub(r'([?&])(key|api_key|token|secret)=[^&]*', r'\1\2=***', url)
    log(f"{method} {safe_url}")

    last_error = None
    rate_limit_count = 0
    for attempt in range(retries):
        try:
            with urllib.request.urlopen(req, timeout=timeout) as response:
                body = response.read().decode('utf-8')
                log(f"Response: {response.status} ({len(body)} bytes)")
                if raw:
                    return body
                return json.loads(body) if body else {}
        except urllib.error.HTTPError as e:
            body = None
            try:
                body = e.read().decode('utf-8')
            except (OSError, UnicodeDecodeError):
                pass
            log(f"HTTP Error {e.code}: {e.reason}")
            if body:
                snippet = " ".join(body.split())
                log(f"Error body: {snippet[:200]}")
            last_error = HTTPError(f"HTTP {e.code}: {e.reason}", e.code, body)

            # Don't retry client errors (4xx) except rate limits
            if 400 <= e.code < 500 and e.code != 429:
                raise last_error

            # Cap 429 retries separately to avoid wasting latency
            if e.code == 429:
                rate_limit_count += 1
                if rate_limit_count >= max_429_retries:
                    raise last_error

            if attempt < retries - 1:
                if e.code == 429:
                    # Respect Retry-After header, fall back to exponential backoff
                    retry_after = e.headers.get("Retry-After") if hasattr(e, 'headers') else None
                    if retry_after:
                        try:
                            delay = float(retry_after)
                        except ValueError:
                            delay = RETRY_DELAY * (2 ** attempt) + 1
                    else:
                        delay = RETRY_DELAY * (2 ** attempt) + 1  # 3s, 5s, 9s...
                    log(f"Rate limited (429). Waiting {delay:.1f}s before retry {attempt + 2}/{retries}")
                else:
                    delay = RETRY_DELAY * (2 ** attempt)
                time.sleep(delay)
        except urllib.error.URLError as e:
            log(f"URL Error: {e.reason}")
            last_error = HTTPError(f"URL Error: {e.reason}")
            if attempt < retries - 1:
                time.sleep(RETRY_DELAY * (attempt + 1))
        except json.JSONDecodeError as e:
            log(f"JSON decode error: {e}")
            last_error = HTTPError(f"Invalid JSON response: {e}")
            raise last_error
        except (OSError, TimeoutError, ConnectionResetError) as e:
            # Handle socket-level errors (connection reset, timeout, etc.)
            log(f"Connection error: {type(e).__name__}: {e}")
            last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
            if attempt < retries - 1:
                time.sleep(RETRY_DELAY * (attempt + 1))

    if last_error:
        raise last_error
    raise HTTPError("Request failed with no error details")


def get(url: str, headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a GET request."""
    return request("GET", url, headers=headers, **kwargs)


def post(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> Dict[str, Any]:
    """Make a POST request with JSON body."""
    return request("POST", url, headers=headers, json_data=json_data, **kwargs)


def post_raw(url: str, json_data: Dict[str, Any], headers: Optional[Dict[str, str]] = None, **kwargs) -> str:
    """Make a POST request with JSON body and return raw text."""
    return request("POST", url, headers=headers, json_data=json_data, raw=True, **kwargs)


def get_reddit_json(path: str, timeout: int = DEFAULT_TIMEOUT, retries: int = MAX_RETRIES) -> Dict[str, Any]:
    """Fetch Reddit thread JSON.

    Args:
        path: Reddit path (e.g., /r/subreddit/comments/id/title)
        timeout: HTTP timeout per attempt in seconds
        retries: Number of retries on failure

    Returns:
        Parsed JSON response
    """
    # Ensure path starts with /
    if not path.startswith('/'):
        path = '/' + path

    # Remove trailing slash and add .json
    path = path.rstrip('/')
    if not path.endswith('.json'):
        path = path + '.json'

    url = f"https://www.reddit.com{path}?raw_json=1"

    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/json",
    }

    return get(url, headers=headers, timeout=timeout, retries=retries)


path: scripts/lib/log.py
content: """Shared logging utilities for last30days skill."""

import os
import sys

DEBUG = os.environ.get("LAST30DAYS_DEBUG", "").lower() in ("1", "true", "yes")


def debug(msg: str) -> None:
    """Log debug message to stderr (only when LAST30DAYS_DEBUG is set)."""
    if DEBUG:
        sys.stderr.write(f"[DEBUG] {msg}\n")
        sys.stderr.flush()


def source_log(prefix: str, msg: str, *, tty_only: bool = True) -> None:
    """Log a source module message to stderr.

    Args:
        prefix: Source label (e.g. "Reddit", "Bird").
        msg: Message text.
        tty_only: If True, only log when stderr is a TTY (avoids cluttering
                  non-interactive output like Claude Code).
    """
    if tty_only and not sys.stderr.isatty():
        return
    sys.stderr.write(f"[{prefix}] {msg}\n")
    sys.stderr.flush()


path: scripts/lib/snippet.py
content: """Best-window extraction for rerankable evidence snippets."""

from __future__ import annotations

from . import relevance, schema


def _truncate_words(text: str, max_words: int) -> str:
    words = text.split()
    if len(words) <= max_words:
        return text.strip()
    return " ".join(words[:max_words]).strip() + "..."


def _windows(words: list[str], size: int, overlap: int) -> list[str]:
    if not words:
        return []
    if len(words) <= size:
        return [" ".join(words)]
    step = max(1, size - overlap)
    return [
        " ".join(words[start:start + size])
        for start in range(0, len(words), step)
    ]


def extract_best_snippet(
    item: schema.SourceItem,
    ranking_query: str,
    max_words: int = 120,
) -> str:
    """Prefer existing snippets, else extract the best matching evidence window."""
    preferred = item.snippet.strip()
    if preferred:
        return _truncate_words(preferred, max_words)

    body = item.body.strip()
    if not body:
        return _truncate_words(item.title, max_words)

    words = body.split()
    candidates = _windows(words, size=min(max_words, 110), overlap=30)
    if not candidates:
        return _truncate_words(body, max_words)

    best = max(
        candidates,
        key=lambda candidate: relevance.token_overlap_relevance(ranking_query, candidate),
    )
    return _truncate_words(best, max_words)


path: scripts/lib/vendor/bird-search/lib/twitter-client-types.js
content: export {};
//# sourceMappingURL=twitter-client-types.js.map

Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)