Skill: dodge1218/batch-executor (version 1.0.0)

--- name: batch-executor version: 1.0.0 description: Full batch processor for corpus-scale task execution. Handles Google Drive dumps, ChatGPT exports, Apple Notes, or any large collection of mixed content (ideas + instructions + references + noise). Classifies items, spawns sub-agents for heavy work, checkpoints progress, and never loses input. Option C of the task management system. Use for 20+ items or when input is a file/folder dump rather than a chat message. --- # Batch Executor Corpus-scale processing: classify → prioritize → spawn → checkpoint → reconcile. Unlike task-extractor (for 3-12 inline tasks) or batch-cognition (for idea analysis), this skill EXECUTES at scale with sub-agent parallelism. ## When to Use - Google Drive folder dump (mixed docs, notes, spreadsheets) - ChatGPT conversation export (3K+ prompts) - Apple Notes dump (years of ideas) - Any input > 20 items or > 10K tokens of raw content - File-based input (not inline chat messages — use task-extractor for those) ## Architecture ``` INPUT (any scale) ↓ PHASE 1: INGEST — save raw to disk, never lose ↓ PHASE 2: CLASSIFY — type each item, estimate effort ↓ PHASE 3: TRIAGE — score by value, group by dependency ↓ PHASE 4: EXECUTE — spawn sub-agents (max 3 concurrent) ↓ PHASE 5: RECONCILE — verify completions, retry failures ↓ PHASE 6: REPORT — value stack, patterns, action items ``` ## Phase 1: INGEST Save ALL raw input to `systems/batch-executor/corpus/YYYY-MM-DD-SOURCE.md` BEFORE any processing. For file inputs: - PDF → extract text via `pdf` tool - CSV/JSON → parse, one item per row/object - Markdown → split on `## ` headers or `---` separators - ChatGPT export → parse `conversations.json`, group by chain_id - Google Drive → process each file, flatten into items Create the manifest: ```markdown # Corpus Manifest: [source] [date] # Total items: [N] # Raw file: [path] # Status: INGESTED | # | First 80 chars | Type | Effort | Status | |---|---------------|------|--------|--------| | 1 | ... | ? | ? | INGESTED | ``` ## Phase 2: CLASSIFY For each item, assign: | Type | Description | Action | |------|-------------|--------| | TASK | Has a clear action verb + deliverable | EXECUTE | | IDEA | Speculative, "what if", product concept | SCORE (ICE) | | REFERENCE | Link, citation, spec, documentation | CATALOG | | DECISION | "We decided X", "going with Y" | RECORD | | HALF_THOUGHT | Fragment, incomplete, trails off | COMPLETE then re-classify | | MODEL_OUTPUT | AI-generated, assistant voice | EXTRACT core idea, discard wrapper | | DUPLICATE | Same as item #X | MERGE | | NOISE | Test, filler, meta-commentary | SKIP | Effort per item: - **TRIVIAL** (< 1 min): file rename, note capture, config change - **QUICK** (1-5 min): web search, small edit, API call - **MEDIUM** (5-30 min): build a page, write a doc, research topic - **HEAVY** (30+ min): full app build, deep research, multi-step workflow - **BLOCKED**: needs human input, credentials, or external dependency Update manifest with Type + Effort columns. ## Phase 3: TRIAGE Score each TASK and IDEA using quick ICE: - **I (Impact)**: 1-5 — how much does this move the needle? - **C (Cost)**: 1-5 — how cheap/fast to do? (inverted: 5 = trivial) - **E (Exploit)**: 1-5 — how quickly does this produce value? - **Score** = I × C × E (max 125) Sort by score descending. Group by dependency chains. Create execution plan: ```markdown # Execution Plan ## Wave 1 (parallel, no dependencies) - Item #14 (ICE: 100) — HEAVY → sub-agent - Item #3 (ICE: 80) — MEDIUM → sub-agent - Item #7 (ICE: 75) — QUICK → inline ## Wave 2 (depends on Wave 1) - Item #9 (depends on #14) — MEDIUM → after #14 completes ## Skip (NOISE/DUPLICATE) - Items #2, #5, #11 — reason: [...] ## Blocked (needs human) - Item #8 — needs API key from Ryan ``` ## Phase 4: EXECUTE Rules: 1. **Max 3 sub-agents concurrent.** Wait for one to complete before spawning another. 2. **QUICK items: execute inline** (no sub-agent overhead for < 5 min tasks). 3. **MEDIUM/HEAVY items: spawn sub-agent** with clear task description + acceptance criteria. 4. **Each sub-agent gets**: the item content, relevant context from other items, and the target artifact path. 5. **Track in manifest**: status → EXECUTING, then ✅ DONE / ❌ FAILED / ⚠️ PARTIAL. Sub-agent spawn template: ``` Task: [item summary] Context: [relevant items from this corpus] Deliverable: [specific file/artifact expected] Acceptance: [how to verify it's done] Workspace: [path] ``` Checkpoint every 5 completed items: - Update manifest - Report to user: "[X]/[N] done. [Y] in progress. Top findings so far: [...]" - If user is idle (no response in 30s), continue - Commit progress to git ## Phase 5: RECONCILE After all waves complete (or all sub-agents return): 1. Re-read manifest 2. For each ❌ FAILED: log reason, decide retry or escalate 3. For each 🔄 sub-agent still running: check status, kill if stale (> 30 min no progress) 4. For each ⚠️ PARTIAL: note what's left 5. Retry failed items once (different approach if possible) ## Phase 6: REPORT Generate final report at `systems/batch-executor/reports/YYYY-MM-DD-SOURCE-report.md`: ```markdown # Corpus Report: [source] # Processed: [date] # Total: [N] items # Results: [done] ✅ | [failed] ❌ | [partial] ⚠️ | [skipped] ⏭️ | [blocked] 🔒 ## Value Stack (top items by impact) 1. [item] — [outcome] — [next step] 2. ... ## Patterns Discovered - [theme or connection across items] ## Action Items (immediate) - [ ] [task from corpus that needs follow-up] ## Parked (valuable but not now) - [item] — reason: [why later] ## Blocked (needs human) - [item] — needs: [what] ## Statistics - Items by type: TASK [x], IDEA [x], REFERENCE [x], NOISE [x] - Items by effort: TRIVIAL [x], QUICK [x], MEDIUM [x], HEAVY [x] - Sub-agents spawned: [x] - Total execution time: [x min] ``` Append to `systems/batch-cognition/value-stack.md` (shared with batch-cognition skill). Log learnings to `.learnings/LEARNINGS.md`. ## Commands `status` — show manifest progress `pause` — stop spawning, let running agents finish `resume` — continue from where we left off (re-read manifest) `skip [#]` — skip item number `retry [#]` — retry failed item `block [#] [reason]` — mark as blocked `priority [#]` — move item to top of queue `done` — trigger report even if items remain ## Key Rules 1. **INGEST FIRST.** Raw content hits disk before ANY processing. 2. **Max 3 concurrent sub-agents.** More = chaos, dropped results, context confusion. 3. **Checkpoint every 5.** Git commit progress. User update. 4. **Never mark ✅ without artifact evidence.** File exists, build passes, URL responds. 5. **NOISE is not failure.** Skipping noise is correct behavior. Report it transparently. 6. **Corpus items cross-reference.** Item #14 may be context for item #27. Pass relevant context to sub-agents. 7. **Resume is first-class.** If session dies, `resume` re-reads manifest and continues from last checkpoint. 8. **ICE scoring is fast.** 30 seconds per item max. Don't overthink triage — execute. ## Integration with Other Skills - **task-extractor**: For inline chat messages (3-12 items). Batch-executor is for file/corpus scale (20+). - **batch-cognition**: For idea analysis (THINK-heavy). Batch-executor is for execution (PLAY-heavy). - **orchestrator**: Batch-executor can be invoked BY the orchestrator when it detects a corpus dump. - **recorder**: After batch-executor completes, route to recorder to update STATUS.md.


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: stale | SkillSpector: suspicious (MEDIUM)