Skill: zhiyuanw101/claw-drive (version 0.4.4)

--- name: claw-drive description: "Claw Drive — AI-managed personal drive for OpenClaw. Auto-categorize, tag, deduplicate, and retrieve files with natural language. Backed by Google Drive for cloud sync and security. Use when receiving files to store, or when asked to find/retrieve a previously stored file." homepage: https://github.com/dissaozw/claw-drive metadata: { "openclaw": { "emoji": "📂", "requires": { "bins": ["claw-drive"] }, "install": [ { "id": "brew", "kind": "brew", "formula": "dissaozw/tap/claw-drive", "bins": ["claw-drive"], "label": "Install Claw Drive (brew)", }, ], }, } --- # Claw Drive Organize and retrieve personal files with auto-categorization and a searchable index. ## ⚠️ Privacy — Read This First **File contents are personal data. Treat them accordingly.** - **NEVER read file contents without explicit user consent.** Always ask first. Always. - **If the user doesn't reply → default to SENSITIVE.** Silence = no consent. - **`identity/` files are ALWAYS sensitive** — never read, never extract, never log contents. - **Extracted content enters the conversation transcript** which is logged permanently to `.jsonl` files. Once you read a file, its contents are in the logs forever. - **Descriptions in INDEX.jsonl are also persistent.** Don't put sensitive details (SSNs, account numbers, passwords) in descriptions even for non-sensitive files — use redacted/partial forms (e.g. "account ending ****4321"). - **When in doubt, don't read.** A vague index entry is better than leaked personal data. **Data locality:** All data stays on your machine. INDEX.jsonl, stored files, and hash ledger are local. Conversation transcripts (`.jsonl`) are also local to your OpenClaw instance. Nothing is sent to external servers unless you explicitly enable Google Drive sync (optional, and only syncs the files you choose). ## Dependencies - **claw-drive CLI** — `brew install dissaozw/tap/claw-drive` (or `make install` from skill directory for manual setup) - **pymupdf** — PDF text extraction (`uv run --with pymupdf` — no global install needed) - **rclone** — Google Drive sync (optional): `brew install rclone` - **fswatch** — file watch daemon (optional): `brew install fswatch` ## ⚠️ CLI Usage — Read This Before Running Anything **ALWAYS use the `claw-drive` CLI. NEVER use `cp`, `mv`, or direct file writes to `~/claw-drive/`.** The CLI handles copying, hashing, deduplication, and index updates atomically. Bypassing it causes: - Files stored without hash registration → dedup breaks silently - INDEX.jsonl out of sync with actual files - Version confusion when replacing files **PATH note:** If installed via Homebrew (`brew install dissaozw/tap/claw-drive`), the binary is in `/opt/homebrew/bin/` and should be in PATH automatically. If installed manually, `~/.local/bin` may not be in the agent shell's PATH — use the full path: ```bash claw-drive store ... ``` If the manual symlink is broken, re-run `make install` from `~/.openclaw/skills/claw-drive/` to fix it. ## Setup ```bash claw-drive init [path] ``` This creates the directory structure, INDEX.jsonl, and hash ledger. Default path: `~/claw-drive`. ## Workflow ### Storing a file When receiving a file (email attachment, Telegram upload, etc.): 1. **Privacy check** — ask the user gracefully if the file contains sensitive/personal data: - Something like: "Should I read the contents to index it better, or would you prefer I keep it private and just use the filename?" - **If user says sensitive**, or **if user doesn't reply** → treat as **sensitive** (default-safe) - **If user confirms it's fine to read** → proceed with full extraction - Files going to `identity/` are **always sensitive** — never read contents - Sensitive flow: classify by filename/metadata only. If that's not enough for a good description, ask the user for a brief description. Never read file contents into the conversation. 2. **Extract** (normal files only) — read file contents: - **PDFs:** extract text via `uv run --with pymupdf python3 -c "import pymupdf; ..."` or use the image tool - **Images:** use the image tool to read/describe contents - **Other formats:** read directly if possible - Pull out key entities: names, dates, amounts, account/policy numbers, addresses, etc. 3. **Classify** — determine the best category from the categories table below 4. **Inspect category structure** — after choosing a category, examine existing subfolders in that category (e.g. with `tree`/`find`) before finalizing destination 5. **Choose destination path** - If an existing subfolder is a clear semantic match, store there - If multiple existing subfolders could match (conflicting/ambiguous), store at category root - Store at category root when the file is only generally related to the category and lacks specific detail - Create a new subfolder only when no existing subfolder fits and the file has clear specific detail that justifies one 6. **Name** — choose a descriptive filename: `<subject>-<detail>-<YYYY-MM-DD>.<ext>` 7. **Describe** — write a rich description using extracted content (or user-provided description for sensitive files). Include key details (dates, amounts, IDs, names) so the file is findable by any relevant search term. Don't be vague — "insurance card" is bad, "Acme Insurance ID cards - 2024 Honda Civic, Policy ****3441, effective 1/21/2026–7/21/2026" is good. 8. **Tag** — include specific tags from extracted content (model names, policy numbers, VINs, entity names) in addition to category tags 9. **Store** — run the CLI (use full path if `claw-drive` not in PATH): ```bash claw-drive store <file> --category <cat> --name 'clean-name.ext' --desc 'Rich description with key details' --tags 'tag1, tag2' --source telegram ``` - **Shell quoting safety:** Prefer single quotes for `--desc`/`--tags`/`--name` when constructing shell commands. This avoids `$` expansion (e.g. currency amounts like `$941.39`) and prevents metadata corruption. ⚠️ **Do NOT use `cp` or write files directly to `~/claw-drive/`.** The CLI is the only correct way to store files — it handles copying, hashing, dedup, and index updates atomically. 10. **Report** — tell the user: path, category, tags, key extracted details, and what was indexed The CLI handles copying, hashing, deduplication, and index updates automatically. If the file is a duplicate, it will be rejected. The `--name` flag lets you override the original filename (which may be ugly like `file_17---8c1ee63d-...`) with a clean, descriptive name. ### Retrieving a file **Do NOT read INDEX.jsonl directly in the main session.** Spawn a search sub-agent instead. This keeps the index out of your context window and scales to large file collections. #### Why sub-agent? The index grows with every stored file (~300 bytes/entry). At 1000+ files, reading the full index into the main agent's context wastes tokens and may hit context limits. A sub-agent runs in its own isolated session with a cheap model, reads the index, and returns only the matching entries. #### How to spawn Use `sessions_spawn` with: - `mode`: `run` - `model`: A lightweight model is recommended (the search task is simple). Resolution order: 1. Explicit `model` param on `sessions_spawn` (if provided) 2. `agents.defaults.subagents.model` in config (if set) 3. Falls back to the main agent's model - `task`: The prompt below, with the user's query filled in ``` You are a file search agent. Read ~/claw-drive/INDEX.jsonl and find entries matching this query: "<USER_QUERY>" Return ONLY valid JSON, no explanation: { "matches": [ { "path": "<path from index>", "desc": "<desc from index>", "date": "<date from index>", "tags": ["<tags from index>"], "confidence": "high|medium|low" } ], "total_indexed": <number of entries in index>, "query": "<original query>" } Rules: - Max 5 matches, sorted by relevance - confidence: high = exact match, medium = likely relevant, low = tangential - If no matches, return {"matches": [], "total_indexed": N, "query": "..."} - Only read INDEX.jsonl, never read file contents ``` #### Receive and deliver 1. The sub-agent auto-announces its result back to your session 2. Parse the JSON from the announce message 3. Prepend `~/claw-drive/` to each `path` to get the full file path 4. **Send the file:** The claw-drive directory may not be in the message tool's allowed paths. If sending fails with "not under an allowed directory", copy the file to a temp location first (e.g. workspace), send it, then clean up: ```bash cp ~/claw-drive/<path> ~/.openclaw/workspace/ # send via message tool rm ~/.openclaw/workspace/<filename> ``` 5. **Never show raw sub-agent JSON to the user.** The announce message is internal — immediately process it and deliver the file. The user should only see the file and a brief description, not search internals. 6. For multiple matches, send the most relevant one and list the rest — let the user pick #### Troubleshooting: pairing required If `sessions_spawn` returns `pairing required`, the sub-agent's exec harness needs device pairing approval. Run: ```bash openclaw devices list # find the pending request openclaw devices approve <request-id> ``` This is a one-time setup — once approved, subsequent spawns work without re-pairing. #### Index format INDEX.jsonl is a JSONL file — one JSON object per line. Each entry has: `date`, `path`, `desc`, `tags` (array), `source`, and optional fields `metadata` (JSON), `original_name`, `correspondent`. ### Updating an entry ```bash claw-drive update <path> --desc "new description" --tags "new, tags" ``` Both `--desc` and `--tags` are optional (at least one required). Uses `jq` for atomic rewrite. ### Deleting a file ```bash claw-drive delete <path> --force ``` Without `--force`, shows what would be deleted (dry run). With `--force`, removes file + index entry + dedup hash. ### Tagging Tags add cross-category searchability. A file lives in one folder but can have multiple tags. **Guidelines:** - 1-5 tags per file, comma-separated - Lowercase, single words or short hyphenated phrases - Always include the category name as a tag (e.g. `medical` for files in `medical/`) - Add cross-cutting tags for things like: entity names (`my-cat`), document type (`invoice`, `receipt`, `report`), context (`emergency`, `tax-2025`) - Reuse existing tags when possible — read INDEX.jsonl to see existing tags before inventing new ones **Examples:** ```bash # Insurance PDF — after extracting: policy number, vehicle, VIN, dates, agent claw-drive store file.pdf -c insurance -n "acme-auto-id-cards.pdf" \ -d "Acme Insurance ID cards - 2024 Honda Civic, VIN 1HGBH41JXMN109186, Policy ****3441, effective 1/21/2026–7/21/2026, agent Jane Smith (555) 123-4567" \ -t "insurance, auto, acme, id-card, honda-civic, california" -s telegram # Vet invoice — after extracting: clinic, amount, diagnosis, pet name claw-drive store invoice.pdf -c medical -n "my-cat-vet-invoice-2026-02-15.pdf" \ -d "VEG emergency visit invoice - Max (cat), $1,234.56, bronchial pattern diagnosis, prednisolone prescribed" \ -t "medical, invoice, max, emergency, vet" -s email # W-2 — after extracting: employer, tax year, wages claw-drive store w2.pdf -c finance -n "w2-2025.pdf" \ -d "W-2 tax form 2025 - Employer: Acme Corp, wages $120,000" \ -t "finance, tax-2025, w2" -s email # Sensitive file — user said "keep it private" or didn't reply claw-drive store scan.pdf -c identity -n "passport-scan-2026.pdf" \ -d "Passport scan" \ -t "identity, passport" -s telegram # Sensitive file — user provided brief description claw-drive store doc.pdf -c contracts -n "apartment-lease-2026.pdf" \ -d "Apartment lease agreement, signed Jan 2026" \ -t "contracts, lease, housing" -s email ``` ### Naming conventions - Lowercase, hyphens between words: `my-cat-vet-invoice-2026-02-15.pdf` - Include date when relevant - Include subject/entity name for clarity - Keep it human-readable — no UUIDs or timestamps ### Categories Categories are **not fixed** — the agent can create any category that makes sense. The CLI does `mkdir -p` automatically. These are the defaults created by `init`, but use whatever fits: | Category | Use for | |----------|---------| | documents | General docs, letters, forms, manuals | | finance | Tax returns, bank statements, investment docs, pay stubs | | insurance | Insurance policies, claims, coverage documents | | medical | Health records, lab results, prescriptions, pet health | | travel | Boarding passes, itineraries, hotel bookings, visas | | identity | Passport scans, birth certs, SSN docs (⚠️ sensitive) | | receipts | Purchase receipts, warranties, service invoices | | contracts | Leases, employment agreements, legal docs | | photos | Personal photos, document scans | | misc | Anything that doesn't fit above | Need `housing/`, `work/`, `pets/`? Just use it — the directory is created on first store. **When in doubt:** `misc/` is fine. Better to store it somewhere than not at all. ## Migration Bulk-import files from an existing directory: ```bash # 1. Scan source directory into a plan claw-drive migrate scan ~/messy-folder plan.json # 2. Agent classifies each file (fills in category, name, tags, description in the JSON) # 3. Review claw-drive migrate summary plan.json # 4. Dry run claw-drive migrate apply plan.json --dry-run # 5. Execute claw-drive migrate apply plan.json ``` The plan JSON contains one entry per file with `category`, `name`, `tags`, `description` fields (initially null). The agent fills these in using the same extract-first approach, then `apply` copies files with full dedup and indexing. ## Sync (Optional) Claw Drive can auto-sync to Google Drive (or any rclone-supported backend) via a background daemon. ### Prerequisites ```bash brew install rclone fswatch ``` ### Authorization Run `claw-drive sync auth`. It opens a browser on the machine for Google sign-in. **What happens:** - rclone requests **Google Drive file access only** (not full Google account) - OAuth token is stored locally at `~/.config/rclone/rclone.conf` — never sent to any third party - Data flows directly from your machine to Google Drive — no intermediary servers - You can revoke access anytime via Google Account → Security → Third-party apps **Agent behavior during auth:** 1. Run `claw-drive sync auth` in background 2. Try the OpenClaw browser tool to click through the Google consent screen 3. If browser tool is unavailable, send the auth URL to the user and ask them to complete sign-in on the machine (e.g. via Screen Sharing) 4. Wait for rclone to capture the token ### Commands ```bash claw-drive sync setup # verify deps and config claw-drive sync start # start background daemon (fswatch + rclone) claw-drive sync stop # stop daemon claw-drive sync push # manual one-shot sync claw-drive sync status # show sync status ``` The daemon watches the drive directory for file changes and syncs to the remote within seconds. It runs as a launchd service — starts on login, restarts on failure. Logs: `~/Library/Logs/claw-drive/sync.log` ### Per-category privacy Use the `exclude` list in `.sync-config` to keep sensitive directories local-only. `identity/` is excluded by default. ## Verify Check index ↔ disk ↔ hash consistency: ```bash claw-drive verify # report issues claw-drive verify --fix # auto-repair what's fixable ``` **Auto-fixable:** missing on disk (removes stale index entry), missing hash (re-registers). **Manual review:** orphan files (no metadata to index), hash mismatches (possible corruption). Run `verify` after manual file operations or when something seems off. ## Tips - The CLI maintains INDEX.jsonl automatically — don't edit it manually - PDF text extraction: `uv run --with pymupdf python3 -c "import pymupdf; ..."` - Use `claw-drive status` to see file counts, size, and sync status ## Privacy Checklist (every store) Before storing any file, verify: - [ ] Did I ask the user about privacy? (not optional) - [ ] If no reply: am I treating it as sensitive? (must be yes) - [ ] If sensitive: am I skipping content extraction? (must be yes) - [ ] If `identity/`: am I skipping extraction regardless? (must be yes) - [ ] Are there SSNs, full account numbers, or passwords in my description? (must be no) - [ ] Would I be comfortable if this INDEX.jsonl entry leaked? (must be yes)

path: CHANGELOG.md
content: # Changelog

All notable changes to this project will be documented in this file.

## [0.2.0] - 2026-02-17

### Added
- `--name` flag for `store` command — specify clean filenames instead of preserving ugly originals
- `insurance` category (policies, ID cards, claims, coverage docs)
- Migration section in README with full workflow documentation
- Test for `--name` flag (37 tests total)

### Fixed
- Test suite no longer clobbers `~/.config/claw-drive/config` (redirects to temp config via env var)
- `sync status` daemon detection uses `launchctl list <name>` directly instead of grep
- SKILL.md rewritten to reference CLI commands instead of manual cp/hash/index workflow
- Stale `claw-drive-sync` references updated to `claw-drive sync <subcommand>`

## [0.1.0] - 2026-02-17

### Added
- Initial release
- CLI with subcommands: `init`, `store`, `search`, `list`, `tags`, `status`, `sync`
- Auto-categorization into 9 file categories
- Tag-based cross-category search via INDEX.md
- SHA-256 content-based deduplication
- Google Drive sync daemon (fswatch + rclone) with launchd integration
- Per-category privacy controls via `.sync-config` exclude list
- JSON output mode (`--json`) for all commands
- OpenClaw skill integration via SKILL.md


path: README.md
content: # 🗄️ Claw Drive

**Google Drive stores your files. Claw Drive understands them.**

[![License: MIT](https://img.shields.io/badge/License-MIT-ffd60a?style=flat-square)](https://opensource.org/licenses/MIT)
[![macOS](https://img.shields.io/badge/macOS-supported-0078d7?logo=apple&logoColor=white&style=flat-square)](https://www.apple.com/macos/)
[![Shell](https://img.shields.io/badge/Shell-bash-4EAA25?logo=gnubash&logoColor=white&style=flat-square)](https://www.gnu.org/software/bash/)
[![CI](https://github.com/dissaozw/claw-drive/actions/workflows/ci.yml/badge.svg)](https://github.com/dissaozw/claw-drive/actions/workflows/ci.yml)

<p align="center">
  <img src="assets/demo-before.png" width="480" alt="Before: navigating 7 layers of folders">
  <br><em>😩 Before — 7 layers deep, 3 "final" versions</em>
</p>

<p align="center">
  <img src="assets/demo-after.png" width="480" alt="After: one message in Telegram, file returned instantly">
  <br><em>✨ After — one sentence, file in hand</em>
</p>

Claw Drive is an AI-managed personal drive. It auto-categorizes your files, tags them for cross-cutting search, deduplicates by content, and retrieves them in natural language — all backed by Google Drive for cloud sync and security.

**Privacy is not a feature — it's the foundation.** Your agent never reads file contents without asking. If you don't respond, it defaults to private. Sensitive categories like `identity/` are never read, never synced. Your data stays yours.

## Features

- 📂 **Auto-categorize** — files sorted into the right folder without you thinking about it
- 🏷️ **Smart tagging** — cross-category search (a vet invoice is both `medical` and `invoice`)
- 🔍 **Natural language retrieval** — "find my cat's vet records" just works
- 🧬 **Content-aware dedup** — SHA-256 hash check prevents storing the same file twice
- ☁️ **Google Drive sync** — optional real-time backup via fswatch + rclone
- 🔒 **Privacy-first** — local-first by default, sensitive categories excluded from sync, default-safe content handling
- 🛡️ **Sensitive file protection** — agent asks before reading contents; defaults to private if no reply
- 📋 **Custom metadata** — attach structured data (expiry dates, policy numbers, amounts) to any file
- 👤 **Correspondent tracking** — record who sent or issued each file
- 🔄 **Reindex** — batch re-enrich old files as your agent gets smarter
- 📛 **Original name tracking** — renames are recorded, both names searchable
- 🤖 **AI-native** — designed for [OpenClaw](https://github.com/openclaw/openclaw) agents, with a CLI under the hood

## Install

### Homebrew (recommended)

```bash
brew install dissaozw/tap/claw-drive
claw-drive init
```

### As an OpenClaw Skill

Clone into your OpenClaw skills directory — OpenClaw picks it up automatically on the next session:

```bash
git clone https://github.com/dissaozw/claw-drive.git ~/.openclaw/skills/claw-drive
cd ~/.openclaw/skills/claw-drive
make install   # symlinks claw-drive to ~/.local/bin (or PREFIX=/usr/local make install)
claw-drive init
```

That's it. Your agent will see the skill and can start storing files immediately.

> **Updating:** `cd ~/.openclaw/skills/claw-drive && git pull`

### Optional: Google Drive Sync

```bash
brew install rclone fswatch   # sync dependencies
claw-drive sync auth          # one-time — opens browser for Google auth
claw-drive sync start         # start background sync daemon
```

### Optional: PDF Extraction

PDF content extraction uses [PyMuPDF](https://pymupdf.readthedocs.io/) via `uv` — no global install needed. It runs automatically when the agent stores a PDF with content reading enabled.

## Usage

Claw Drive is designed to be used through your AI agent. You don't organize files — your agent does.

### Storing files

Send a file to your agent (Telegram, email, etc.) and it handles everything:

1. **Asks about privacy** — "Should I read the contents, or keep it private?"
2. **Extracts content** (if permitted) — reads PDFs, images, docs to pull out key details
3. **Categorizes** the file into the right folder
4. **Names** it with a descriptive, date-stamped filename
5. **Checks for duplicates** by content hash
6. **Tags** it for cross-category search with specific identifiers
7. **Indexes** it in INDEX.jsonl with a rich, searchable description
8. **Reports** back what it did

> 📎 *"Here's my auto insurance card"*
>
> 🔒 *"Should I read the contents to index it better, or keep it private?"*
>
> 👤 *"Go ahead"*
>
> ✅ Stored: `insurance/acme-auto-id-cards.pdf`
> Policy ****3441 · 2024 Honda Civic · Effective 1/21/2026–7/21/2026
> Tags: insurance, auto, acme, honda-civic, california

If you don't reply or say it's sensitive, the agent classifies by filename only and asks for a brief description if needed. Your data is never read without consent.

### Retrieving files

Just ask in natural language:

> *"Find my cat's medical records"*
> *"Show me all invoices from January"*
> *"Do I have a copy of my W-2?"*

The agent reads INDEX.jsonl directly — its semantic understanding beats any grep. It finds files by meaning, not string matching.

### What you never have to do

- Pick a folder
- Think of a filename
- Remember where you put something
- Manually organize anything

## CLI Reference

The CLI handles **write operations** — store, sync, migrate — where atomicity matters (dedup + index updates). For **read operations** (search, list, tags), the agent reads INDEX.jsonl directly.

| Command | Description |
|---------|-------------|
| `claw-drive init` | Initialize drive directory and INDEX.jsonl |
| `claw-drive store <file> [opts]` | Store a file with categorization, tags, dedup, rename (`--name`), metadata (`--metadata`), correspondent (`--correspondent`) |
| `claw-drive update <path> [opts]` | Update description, tags, metadata, correspondent, and/or source on an existing entry |
| `claw-drive delete <path> [--force]` | Delete a file, its index entry, and dedup hash |
| `claw-drive rm <path> [--force]` | Alias for `delete` |
| `claw-drive status` | Show drive status (files, size, sync) |
| `claw-drive sync auth` | Authorize Google Drive (one-time, opens browser) |
| `claw-drive sync setup` | Check sync dependencies and config |
| `claw-drive sync start` | Start background sync daemon |
| `claw-drive sync stop` | Stop sync daemon |
| `claw-drive sync push` | Manual one-shot sync |
| `claw-drive sync status` | Show sync daemon state |
| `claw-drive reindex scan [--output plan.json]` | Scan drive for orphans + export existing entries for re-enrichment |
| `claw-drive reindex apply <plan.json> [--dry-run]` | Apply enriched reindex plan (add orphans, update existing) |
| `claw-drive migrate scan <dir> [plan.json]` | Scan a directory into a migration plan |
| `claw-drive migrate summary [plan.json]` | Show migration plan breakdown |
| `claw-drive migrate apply [plan.json] [--dry-run]` | Execute migration plan |
| `claw-drive version` | Show version |

## Sync

Optional real-time sync to Google Drive (or any rclone backend). Files sync within seconds of any change. Sensitive directories stay local-only.

See [docs/sync.md](docs/sync.md) for details.

## Migration

Got a messy folder full of unsorted files? Claw Drive's migration workflow handles it:

```bash
# 1. Scan the source directory
claw-drive migrate scan ~/messy-folder migration-plan.json

# 2. AI agent classifies each file (fills in category, name, tags, description)

# 3. Review the plan
claw-drive migrate summary migration-plan.json

# 4. Dry run first
claw-drive migrate apply migration-plan.json --dry-run

# 5. Execute
claw-drive migrate apply migration-plan.json
```

The scan outputs a JSON plan with file metadata (path, size, mime type, extension). The agent fills in classification fields, then `apply` copies files into Claw Drive with full dedup, indexing, and tagging.

## Custom Metadata

Store structured data alongside your files — expiry dates, policy numbers, amounts, anything the agent can answer questions about without reading the original file.

```bash
# Add metadata when storing
claw-drive store insurance-card.pdf -c insurance -d "Auto insurance" \
  --metadata '{"policy":"****3441","expiry":"2026-08","provider":"Farmers"}'

# Add metadata to existing files
claw-drive update insurance/card.pdf --metadata '{"deductible":"$500"}'
```

Metadata merges on update — existing fields are preserved, new fields are added. The agent can now answer "when does my insurance expire?" directly from the index, without opening the file.

## Correspondent Tracking

Track who sent or issued each file — the person, company, or organization it came from:

```bash
# Set on store
claw-drive store invoice.pdf -c finance -d "Q4 invoice" --correspondent "Acme Corp"

# Add to existing file
claw-drive update finance/invoice.pdf --correspondent "Acme Corp"
```

This lets the agent answer questions like "show me everything from Farmers Insurance" or "what did VEG send me?" by filtering on correspondent.

## Reindex

Already have files in Claw Drive but want richer descriptions, better tags, or custom metadata? The reindex workflow lets the agent re-analyze everything:

```bash
# 1. Scan — exports a plan with all files + current index entries
claw-drive reindex scan --output reindex-plan.json

# 2. Agent enriches the plan:
#    - Orphan files: fills in desc, tags, source, metadata
#    - Existing entries: adds new_desc, new_tags, new_metadata to update

# 3. Preview changes
claw-drive reindex apply reindex-plan.json --dry-run

# 4. Apply
claw-drive reindex apply reindex-plan.json
```

As your agent gets smarter, your old files benefit too.

## Original Filename Preservation

When you rename a file on store (`--name`), Claw Drive records the original filename in the index. This means you can search by either name:

```bash
claw-drive store messy-scan-001.pdf -c medical --name "blood-work-2026-02.pdf" -d "..."
# Index records: original_name: "messy-scan-001.pdf"
# Searchable by both "messy-scan" and "blood-work"
```

## Architecture

```
You ← natural language → AI Agent (OpenClaw)
                              │
                        claw-drive CLI
                              │
                        ~/claw-drive/        ← local, source of truth
                              │
                        fswatch + rclone     ← optional real-time sync
                              │
                        Google Drive          ← cloud backup
```

## Categories

Categories are open-ended — agents create new ones as needed. These are the defaults:

| Category | Use for |
|----------|---------|
| `documents/` | General docs, letters, forms, manuals |
| `finance/` | Tax returns, bank statements, pay stubs |
| `insurance/` | Policies, ID cards, claims, coverage docs |
| `medical/` | Health records, prescriptions, pet health |
| `travel/` | Boarding passes, itineraries, visas |
| `identity/` | ID scans, certificates (⚠️ sensitive — excluded from sync) |
| `receipts/` | Purchase receipts, warranties, invoices |
| `contracts/` | Leases, employment, legal agreements |
| `photos/` | Personal photos, document scans |
| `misc/` | Anything that doesn't fit above |

## Privacy & Security

**Claw Drive treats your files as personal data by default.** This isn't an afterthought — it's a core design decision.

### The Problem

AI agents that read your files put those contents into conversation transcripts — which are logged permanently. A "helpful" agent that reads your passport scan, tax return, or medical record has now copied that data into a `.jsonl` log file. That's a leak, not a feature.

### The Solution

Claw Drive's agent **always asks before reading**. And if you don't answer, it assumes the answer is no.

| Scenario | Behavior |
|----------|----------|
| User says "go ahead" | Full content extraction → rich description + specific tags |
| User says "keep it private" | Filename-only classification, asks for brief description |
| **User doesn't reply** | **Defaults to sensitive** — no content reading |
| **File goes to `identity/`** | **Always sensitive** — contents never read, never synced |

### What "sensitive" means in practice

- File contents are **never read** by the agent
- Classification uses **filename and user input only**
- INDEX.jsonl descriptions are kept **generic** (no SSNs, account numbers, etc.)
- `identity/` is **excluded from cloud sync** by default
- The file is still stored, hashed (for dedup), tagged, and indexed — just without content extraction

### Defense in depth

| Layer | Protection |
|-------|-----------|
| Consent | Agent asks before reading any file |
| Default-safe | No reply = sensitive |
| Category rules | `identity/` always sensitive, excluded from sync |
| Sync exclusion | `.sync-config` exclude list for any category |
| Index hygiene | No raw sensitive data in descriptions |
| Local-first | Cloud sync is optional, not default |

## Documentation

- [Tags](docs/tags.md) — tagging guidelines and examples
- [Sync](docs/sync.md) — Google Drive sync setup and daemon
- [Security](docs/security.md) — threat model and privacy controls

## Roadmap

- [x] `update` command — modify description/tags on existing entries
- [x] `delete` command — remove files with atomic index cleanup
- [x] `verify --fix` — self-healing integrity checks
- [x] `reindex` — batch re-enrichment of existing files
- [x] Custom metadata fields — structured data per file
- [x] Correspondent tracking — source person/organization per file
- [x] Original filename preservation on rename
- [ ] Watch folder ingestion (auto-import from Downloads)
- [ ] Encrypted storage for sensitive categories
- [ ] Linux support (inotifywait + systemd)
- [ ] Web dashboard for browsing and search
- [ ] Homebrew tap distribution

## License

[MIT](LICENSE)


path: docs/security.md
content: # Security

Claw Drive is local-first. Your files live on your machine. Cloud sync is optional.

## Threat Model

| Threat | Mitigation |
|--------|-----------|
| Cloud provider breach | Sensitive dirs excluded from sync via `.sync-config` |
| Local disk theft | Use FileVault (macOS) for full-disk encryption |
| File tampering | SHA-256 hashes in `.hashes` detect content changes |
| Accidental deletion | Google Drive versioning + local Time Machine |

## Sensitive Categories

The `identity/` category holds passport scans, SSN docs, and other high-sensitivity files. By default, the example `.sync-config` excludes it from sync:

```yaml
exclude:
  - identity/
```

These files are fully functional locally — indexed, tagged, searchable — but never leave the machine.

## INDEX.md

The index contains file descriptions but no file contents. Keep descriptions of sensitive files brief:

```
| 2026-01-01 | identity/passport-scan.pdf | Passport scan | identity, passport | manual |
```

Don't include account numbers, SSNs, or other PII in descriptions.

## Credentials

- **rclone config** lives at `~/.config/rclone/rclone.conf` — standard location, not managed by Claw Drive
- No API keys or tokens are stored in the drive directory
- No credentials are ever written to INDEX.md or log files


path: docs/sync.md
content: # Sync

Claw Drive can auto-sync to Google Drive (or any rclone-supported backend) via a background daemon.

## Architecture

```
~/claw-drive/  (local, source of truth)
      │
      ├── fswatch (file watcher)
      │       │
      │       └── rclone sync → Google Drive (cloud backup)
      │
      └── launchd (keeps daemon alive)
```

Local is always the source of truth. Cloud sync is optional backup + cross-device access.

## Prerequisites

```bash
brew install rclone fswatch
rclone config  # set up a remote (e.g. "gdrive")
```

## Configuration

Create `~/claw-drive/.sync-config`:

```yaml
backend: google-drive
remote: gdrive:claw-drive
exclude:
  - identity/
  - .hashes
```

### Fields

| Field | Description |
|-------|-------------|
| `backend` | Backend name (informational) |
| `remote` | rclone remote:path |
| `exclude` | List of paths/patterns to never sync |

## Commands

```bash
claw-drive sync setup    # verify deps and config
claw-drive sync start    # start background daemon
claw-drive sync stop     # stop daemon
claw-drive sync push     # manual one-shot sync
claw-drive sync status   # show daemon state + last sync time
```

## Daemon

`claw-drive sync start` installs a launchd service (`com.claw-drive.sync`) that:

1. Watches `~/claw-drive/` with fswatch
2. On any file change, waits 3 seconds (debounce)
3. Runs `rclone sync` to the configured remote
4. Records the sync timestamp in `.sync-state`

The daemon starts on login and restarts on failure.

**Logs:** `~/Library/Logs/claw-drive/sync.log`

## Privacy

Use the `exclude` list to keep sensitive directories local-only:

```yaml
exclude:
  - identity/     # never leaves the machine
  - .hashes       # internal ledger
```

Files in excluded directories are fully functional locally — they just don't sync.


path: docs/tags.md
content: # Tags

Tags enable cross-category search. A file lives in one folder but can have multiple tags.

## How It Works

Tags are stored in the Tags column of `INDEX.md`:

```
| Date | Path | Description | Tags | Source |
|------|------|-------------|------|--------|
| 2026-02-15 | medical/sorbet-vet-invoice.pdf | VEG emergency | medical, invoice, sorbet, emergency | email |
```

Searching by any tag returns matching files:

```bash
claw-drive search sorbet
claw-drive search invoice
claw-drive tags            # list all tags with usage counts
```

## Guidelines

- **1-5 tags** per file, comma-separated
- **Lowercase**, single words or short hyphenated phrases
- **Always include** the category name as a tag (e.g. `medical`)
- **Add cross-cutting tags** for: entity names (`sorbet`), document type (`invoice`, `receipt`), context (`emergency`, `tax-2025`)
- **Reuse existing tags** — run `claw-drive tags` before inventing new ones

## Examples

| File | Tags |
|------|------|
| Vet invoice in `medical/` | `medical, invoice, sorbet, emergency` |
| W-2 in `finance/` | `finance, tax-2025` |
| Trip itinerary in `travel/` | `travel, japan` |
| Lease agreement in `contracts/` | `contracts, lease, apartment` |


path: lib/config.sh
content: #!/usr/bin/env bash
# lib/config.sh — Configuration management for Claw Drive

CLAW_DRIVE_CONFIG_FILE="${CLAW_DRIVE_CONFIG_FILE:-$HOME/.config/claw-drive/config}"

# Load saved drive path from config, env override takes priority
if [[ -n "${CLAW_DRIVE_DIR:-}" ]]; then
  : # env override
elif [[ -f "$CLAW_DRIVE_CONFIG_FILE" ]]; then
  CLAW_DRIVE_DIR=$(cat "$CLAW_DRIVE_CONFIG_FILE")
else
  CLAW_DRIVE_DIR="$HOME/claw-drive"
fi
CLAW_DRIVE_INDEX="$CLAW_DRIVE_DIR/INDEX.jsonl"
CLAW_DRIVE_HASHES="$CLAW_DRIVE_DIR/.hashes"
CLAW_DRIVE_SYNC_CONFIG="$CLAW_DRIVE_DIR/.sync-config"
CLAW_DRIVE_SYNC_STATE="$CLAW_DRIVE_DIR/.sync-state"
CLAW_DRIVE_LOG_DIR="$HOME/Library/Logs/claw-drive"
CLAW_DRIVE_PLIST_NAME="com.claw-drive.sync"
CLAW_DRIVE_PLIST_PATH="$HOME/Library/LaunchAgents/$CLAW_DRIVE_PLIST_NAME.plist"

CLAW_DRIVE_CATEGORIES=(
  documents finance insurance medical travel identity
  receipts contracts photos misc
)

# Ensure drive directory and essential files exist
claw_drive_init() {
  if [[ ! -d "$CLAW_DRIVE_DIR" ]]; then
    echo "❌ Drive directory not found: $CLAW_DRIVE_DIR"
    echo "   Run: claw-drive init"
    return 1
  fi
  [[ -f "$CLAW_DRIVE_HASHES" ]] || touch "$CLAW_DRIVE_HASHES"
  return 0
}

# Parse sync config: get a top-level key value
sync_config_get() {
  local key="$1"
  if [[ -f "$CLAW_DRIVE_SYNC_CONFIG" ]]; then
    grep -E "^${key}:" "$CLAW_DRIVE_SYNC_CONFIG" | sed "s/^${key}:[[:space:]]*//" | tr -d '"'"'"
  fi
}

# Parse sync config: get exclude list
sync_config_excludes() {
  if [[ -f "$CLAW_DRIVE_SYNC_CONFIG" ]]; then
    sed -n '/^exclude:/,/^[^ -]/p' "$CLAW_DRIVE_SYNC_CONFIG" | grep -E '^\s*-' | sed 's/^[[:space:]]*-[[:space:]]*//'
  fi
}

# Build rclone exclude arguments from sync config as a safe array.
# Output format: one arg per line ("--exclude" then pattern), for mapfile consumption.
sync_build_exclude_args_lines() {
  while IFS= read -r pattern; do
    [[ -n "$pattern" ]] || continue
    printf '%s\n' "--exclude" "$pattern"
  done < <(sync_config_excludes)

  # Always exclude internal state files
  printf '%s\n' "--exclude" ".sync-config" "--exclude" ".sync-state"
}


path: lib/dedup.sh
content: #!/usr/bin/env bash
# lib/dedup.sh — Content-based deduplication for Claw Drive

# Hash a file and return SHA-256
dedup_hash() {
  local file="$1"
  shasum -a 256 "$file" | awk '{print $1}'
}

# Check if a file is a duplicate. Returns 0 if duplicate, 1 if new.
# Prints the existing path if duplicate.
dedup_check() {
  local file="$1"

  if [[ ! -f "$CLAW_DRIVE_HASHES" ]]; then
    return 1
  fi

  local hash
  hash=$(dedup_hash "$file")

  local existing
  existing=$(grep "^$hash " "$CLAW_DRIVE_HASHES" | head -1)
  # Preserve paths with spaces by splitting on the double-space delimiter.
  existing="${existing#*  }"

  if [[ -n "$existing" ]]; then
    echo "$existing"
    return 0
  fi

  return 1
}

# Register a file hash in the dedup ledger
dedup_register() {
  local file="$1"
  local relative_path="$2"

  local hash
  hash=$(dedup_hash "$file")

  echo "$hash  $relative_path" >> "$CLAW_DRIVE_HASHES"
}

# Update a registered path in hash ledger after move/rename
# Usage: dedup_move <old_path> <new_path>
dedup_move() {
  local old_path="$1"
  local new_path="$2"

  if [[ ! -f "$CLAW_DRIVE_HASHES" ]]; then
    return 0
  fi

  local tmp
  tmp=$(mktemp) || { echo "❌ mktemp failed" >&2; return 1; }

  while IFS= read -r line || [[ -n "$line" ]]; do
    local hash="${line%%  *}"
    local path="${line#*  }"
    if [[ "$path" == "$old_path" ]]; then
      printf '%s  %s\n' "$hash" "$new_path"
    else
      printf '%s\n' "$line"
    fi
  done < "$CLAW_DRIVE_HASHES" > "$tmp"

  mv "$tmp" "$CLAW_DRIVE_HASHES"
}

# Show dedup stats
dedup_status() {
  local format="${1:-table}"

  if [[ ! -f "$CLAW_DRIVE_HASHES" ]]; then
    echo "No hash ledger found."
    return 0
  fi

  local count
  count=$(wc -l < "$CLAW_DRIVE_HASHES" | xargs)

  if [[ "$format" == "json" ]]; then
    printf '{"tracked_files":%d,"ledger":"%s"}\n' "$count" "$CLAW_DRIVE_HASHES"
  else
    echo "Dedup ledger: $CLAW_DRIVE_HASHES"
    echo "Tracked files: $count"
  fi
}


path: lib/index.sh
content: #!/usr/bin/env bash
# lib/index.sh — INDEX.jsonl management for Claw Drive
#
# INDEX.jsonl is a structured JSONL file — one JSON object per line.
# Agents read it directly for search/list/tag operations.
# This library handles atomic write operations (add, update, delete).

# Safe mktemp wrapper — exits on failure instead of proceeding with empty path
safe_mktemp() {
  local tmp
  tmp=$(mktemp) || { echo "❌ mktemp failed" >&2; return 1; }
  echo "$tmp"
}

# Append a new entry to the index
# Usage: index_add <date> <path> <desc> <tags> <source> [metadata_json] [original_name] [correspondent]
index_add() {
  local date="$1" path="$2" desc="$3" tags="$4" source="$5" metadata="${6:-}" original_name="${7:-}" correspondent="${8:-}"

  # Convert comma-separated tags to JSON array
  local tags_json
  tags_json=$(printf '%s' "$tags" | jq -R 'split(",") | map(gsub("^\\s+|\\s+$";""))')

  local jq_args=(
    --arg date "$date"
    --arg path "$path"
    --arg desc "$desc"
    --argjson tags "$tags_json"
    --arg source "$source"
  )
  local jq_expr='{date:$date, path:$path, desc:$desc, tags:$tags, source:$source}'

  if [[ -n "$metadata" ]]; then
    jq_args+=(--argjson metadata "$metadata")
    jq_expr='{date:$date, path:$path, desc:$desc, tags:$tags, source:$source, metadata:$metadata}'
  fi

  # Build optional fields dynamically
  local optional_fields=""
  if [[ -n "$original_name" ]]; then
    jq_args+=(--arg original_name "$original_name")
    optional_fields="$optional_fields, original_name:\$original_name"
  fi
  if [[ -n "$correspondent" ]]; then
    jq_args+=(--arg correspondent "$correspondent")
    optional_fields="$optional_fields, correspondent:\$correspondent"
  fi

  # Rebuild expression with optional fields
  if [[ -n "$metadata" ]]; then
    jq_expr="{date:\$date, path:\$path, desc:\$desc, tags:\$tags, source:\$source, metadata:\$metadata${optional_fields}}"
  else
    jq_expr="{date:\$date, path:\$path, desc:\$desc, tags:\$tags, source:\$source${optional_fields}}"
  fi

  jq -cn "${jq_args[@]}" "$jq_expr" >> "$CLAW_DRIVE_INDEX"
}

# Remove an entry by path (exact match)
index_remove() {
  local target_path="$1"
  local tmp
  tmp=$(safe_mktemp) || return 1

  jq -c --arg path "$target_path" 'select(.path != $path)' "$CLAW_DRIVE_INDEX" > "$tmp"
  mv "$tmp" "$CLAW_DRIVE_INDEX"
}

# Update fields on an entry by path (exact match)
# Usage: index_update <path> [--desc <desc>] [--tags <tags>]
index_update() {
  local target_path="$1"
  shift

  local new_desc="" new_tags="" new_metadata="" new_correspondent="" new_source=""
  while [[ $# -gt 0 ]]; do
    case "$1" in
      --desc|-d) new_desc="$2"; shift 2 ;;
      --tags|-t) new_tags="$2"; shift 2 ;;
      --metadata|-m) new_metadata="$2"; shift 2 ;;
      --correspondent) new_correspondent="$2"; shift 2 ;;
      --source|-s) new_source="$2"; shift 2 ;;
      *) shift ;;
    esac
  done

  local tmp
  tmp=$(safe_mktemp) || return 1

  local jq_filter='if .path == $path then'
  local jq_args
  jq_args=(--arg path "$target_path")

  if [[ -n "$new_desc" ]]; then
    jq_filter="$jq_filter .desc = \$desc |"
    jq_args+=(--arg desc "$new_desc")
  fi

  if [[ -n "$new_tags" ]]; then
    local tags_json
    tags_json=$(printf '%s' "$new_tags" | jq -R 'split(",") | map(gsub("^\\s+|\\s+$";""))')
    jq_filter="$jq_filter .tags = \$tags |"
    jq_args+=(--argjson tags "$tags_json")
  fi

  if [[ -n "$new_metadata" ]]; then
    # Merge new metadata into existing (or create if absent)
    jq_filter="$jq_filter .metadata = ((.metadata // {}) * \$meta) |"
    jq_args+=(--argjson meta "$new_metadata")
  fi

  if [[ -n "$new_correspondent" ]]; then
    jq_filter="$jq_filter .correspondent = \$correspondent |"
    jq_args+=(--arg correspondent "$new_correspondent")
  fi

  if [[ -n "$new_source" ]]; then
    jq_filter="$jq_filter .source = \$new_source |"
    jq_args+=(--arg new_source "$new_source")
  fi

  # Remove trailing pipe if present
  jq_filter="${jq_filter% |}"
  jq_filter="$jq_filter else . end"

  jq -c "${jq_args[@]}" "$jq_filter" "$CLAW_DRIVE_INDEX" > "$tmp"
  mv "$tmp" "$CLAW_DRIVE_INDEX"
}

# Check if a path exists in the index
index_has() {
  local target_path="$1"
  # jq -e can return non-zero with JSONL streams in some envs despite a match;
  # use exact path membership check for robust existence probing.
  jq -r '.path // empty' "$CLAW_DRIVE_INDEX" 2>/dev/null | grep -Fxq "$target_path"
}

# Rename/move an entry path in the index (exact match)
# Usage: index_move <old_path> <new_path>
index_move() {
  local old_path="$1"
  local new_path="$2"
  local tmp
  tmp=$(safe_mktemp) || return 1

  jq -c --arg old "$old_path" --arg new "$new_path" \
    'if .path == $old then .path = $new else . end' \
    "$CLAW_DRIVE_INDEX" > "$tmp"
  mv "$tmp" "$CLAW_DRIVE_INDEX"
}

# Dedup: remove hash entry for a path (exact match, regex-safe)
dedup_unregister() {
  local target_path="$1"

  if [[ ! -f "$CLAW_DRIVE_HASHES" ]]; then
    return 0
  fi

  local tmp
  tmp=$(safe_mktemp) || return 1
  # Use shell parameter expansion to extract path from "hash  path" format,
  # avoiding unescaped grep which breaks on regex special chars (., (, ), etc.)
  while IFS= read -r line || [[ -n "$line" ]]; do
    local line_path="${line#*  }"
    [[ "$line_path" != "$target_path" ]] && printf '%s\n' "$line"
  done < "$CLAW_DRIVE_HASHES" > "$tmp"
  mv "$tmp" "$CLAW_DRIVE_HASHES"
}


path: lib/migrate.sh
content: #!/usr/bin/env bash
# lib/migrate.sh — Migration: scan arbitrary directories into Claw Drive

# Scan a source directory and output migration-plan.json
migrate_scan() {
  local source_dir="$1"
  local output="${2:-migration-plan.json}"

  if [[ ! -d "$source_dir" ]]; then
    echo "❌ Directory not found: $source_dir"
    return 1
  fi

  # Resolve absolute path
  source_dir="$(cd "$source_dir" && pwd)"

  echo "🔍 Scanning $source_dir ..."

  local files=()
  local count=0

  # Start JSON
  printf '{\n  "source": "%s",\n  "scanned_at": "%s",\n  "files": [\n' \
    "$source_dir" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$output"

  local first=true

  while IFS= read -r -d '' filepath; do
    # Skip hidden files and directories
    [[ "$(basename "$filepath")" == .* ]] && continue

    local relative="${filepath#$source_dir/}"
    local size
    size=$(stat -f%z "$filepath" 2>/dev/null || stat -c%s "$filepath" 2>/dev/null || echo 0)
    local modified
    modified=$(stat -f%Sm -t%Y-%m-%d "$filepath" 2>/dev/null || stat -c%y "$filepath" 2>/dev/null | cut -d' ' -f1 || echo "unknown")
    local ext="${filepath##*.}"
    [[ "$ext" == "$filepath" ]] && ext=""
    local mime
    mime=$(file --brief --mime-type "$filepath" 2>/dev/null || echo "unknown")

    [[ "$first" == "true" ]] || printf ',\n' >> "$output"

    # Escape strings for JSON
    local escaped_path
    escaped_path=$(printf '%s' "$relative" | sed 's/\\/\\\\/g; s/"/\\"/g')
    local escaped_mime
    escaped_mime=$(printf '%s' "$mime" | sed 's/\\/\\\\/g; s/"/\\"/g')

    cat >> "$output" <<EOF
    {
      "source_path": "$escaped_path",
      "size": $size,
      "modified": "$modified",
      "extension": "$ext",
      "mime": "$escaped_mime",
      "category": null,
      "name": null,
      "tags": null,
      "description": null,
      "confidence": null,
      "status": "pending"
    }
EOF
    first=false
    ((count++)) || true

    # Progress every 50 files
    if (( count % 50 == 0 )); then
      echo "   ... $count files scanned"
    fi

  done < <(find "$source_dir" -type f -print0 | sort -z)

  # Close JSON
  printf '\n  ]\n}\n' >> "$output"

  echo "✅ Scanned $count files → $output"
}

# Apply a migration plan
migrate_apply() {
  local plan_file="$1"
  local dry_run="${2:-false}"

  if [[ ! -f "$plan_file" ]]; then
    echo "❌ Plan file not found: $plan_file"
    return 1
  fi

  claw_drive_init || return 1

  echo "📦 Applying migration plan: $plan_file"
  [[ "$dry_run" == "true" ]] && echo "   (dry run — no files will be moved)"
  echo ""

  local source_dir
  source_dir=$(python3 - "$plan_file" <<'PY'
import json, sys
with open(sys.argv[1], 'r', encoding='utf-8') as f:
    print(json.load(f)['source'])
PY
)

  local total=0 stored=0 skipped=0 dupes=0 errors=0

  # Process each file in the plan
  # Note: use process substitution (< <(...)) instead of a pipe so the
  # while-loop runs in the current shell and counter variables persist.
  while IFS=$'\t' read -r src_path category new_name tags description confidence; do
    ((total++)) || true

    if [[ -z "$category" || -z "$new_name" ]]; then
      echo "  ⚠️  Skipping (no category/name): $src_path"
      ((skipped++)) || true
      continue
    fi

    # Validate path components from plan file
    if ! validate_path_component "category" "$category" 2>&1 || \
       ! validate_filename "name" "$new_name" 2>&1; then
      echo "  ❌ Unsafe path in plan: $category/$new_name"
      ((errors++)) || true
      continue
    fi

    # Validate source path from plan: must stay within source_dir
    if ! validate_path_component "source_path" "$src_path" 2>&1; then
      echo "  ❌ Unsafe source path in plan: $src_path"
      ((errors++)) || true
      continue
    fi

    local full_source="$source_dir/$src_path"
    local resolved_source source_root
    resolved_source=$(cd "$(dirname "$full_source")" 2>/dev/null && pwd -P)/$(basename "$full_source")
    source_root=$(cd "$source_dir" 2>/dev/null && pwd -P)
    if [[ "$resolved_source" != "$source_root"/* ]]; then
      echo "  ❌ Source path escapes migration source root: $src_path"
      ((errors++)) || true
      continue
    fi

    if [[ ! -f "$resolved_source" ]]; then
      echo "  ❌ Source missing: $src_path"
      ((errors++)) || true
      continue
    fi

    # Dedup check
    local existing
    if existing=$(dedup_check "$resolved_source"); then
      echo "  🔁 Duplicate (exists at $existing): $src_path"
      ((dupes++)) || true
      continue
    fi

    local dest="$CLAW_DRIVE_DIR/$category/$new_name"

    if [[ "$dry_run" == "true" ]]; then
      echo "  📄 $src_path → $category/$new_name [$tags]"
    else
      mkdir -p "$CLAW_DRIVE_DIR/$category"
      if ! validate_in_drive_dir "$dest"; then
        ((errors++)) || true
        continue
      fi
      cp "$resolved_source" "$dest"
      dedup_register "$dest" "$category/$new_name"

      # Update index
      local date_str
      date_str=$(date +%Y-%m-%d)
      index_add "$date_str" "$category/$new_name" "$description" "$tags" "migration"

      echo "  ✅ $src_path → $category/$new_name"
    fi
    ((stored++)) || true
  done < <(python3 - "$plan_file" <<'PY'
import json, sys
with open(sys.argv[1], 'r', encoding='utf-8') as fh:
    plan = json.load(fh)
for f in plan['files']:
    if f.get('status') == 'skip':
        continue
    cat = f.get('category') or ''
    name = f.get('name') or ''
    tags = f.get('tags') or ''
    desc = f.get('description') or ''
    src = f.get('source_path') or ''
    conf = f.get('confidence') or ''
    print(f'{src}\t{cat}\t{name}\t{tags}\t{desc}\t{conf}')
PY
)

  echo ""
  echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
  echo "Migration complete."
  echo "  Stored: $stored"
  echo "  Duplicates: $dupes"
  echo "  Skipped: $skipped"
  echo "  Errors: $errors"
}

# Show plan summary
migrate_summary() {
  local plan_file="$1"

  if [[ ! -f "$plan_file" ]]; then
    echo "❌ Plan file not found: $plan_file"
    return 1
  fi

  python3 - "$plan_file" <<'PY'
import json, sys
with open(sys.argv[1], 'r', encoding='utf-8') as fh:
    plan = json.load(fh)
files = plan['files']
total = len(files)
ready = sum(1 for f in files if f.get('category') and f.get('name'))
pending = sum(1 for f in files if not f.get('category') or not f.get('name'))
skip = sum(1 for f in files if f.get('status') == 'skip')

print(f'📋 Migration Plan: {plan["source"]}')
print(f'   Scanned: {plan.get("scanned_at", "unknown")}')
print(f'   Total files: {total}')
print(f'   Ready: {ready}')
print(f'   Pending classification: {pending}')
print(f'   Marked skip: {skip}')
print()

# Category breakdown
cats = {}
for f in files:
    c = f.get('category') or 'unclassified'
    cats[c] = cats.get(c, 0) + 1
print('   Categories:')
for c in sorted(cats, key=cats.get, reverse=True):
    print(f'     {c}: {cats[c]}')
PY
}


path: lib/reindex.sh
content: #!/usr/bin/env bash
# lib/reindex.sh — Reindex support for Claw Drive
#
# Scan drive for orphan files and existing entries,
# output a plan for the agent to enrich, then apply updates.

reindex_scan() {
  local output="reindex-plan.json"

  while [[ $# -gt 0 ]]; do
    case "$1" in
      --output|-o) output="$2"; shift 2 ;;
      *) shift ;;
    esac
  done

  claw_drive_init || return 1

  echo "🔍 Scanning $CLAW_DRIVE_DIR for reindex..."

  # Collect indexed paths
  local indexed_paths_file
  indexed_paths_file=$(safe_mktemp) || return 1
  jq -r '.path' "$CLAW_DRIVE_INDEX" > "$indexed_paths_file" 2>/dev/null || true

  # Build orphans array
  local orphans_file
  orphans_file=$(safe_mktemp) || return 1
  echo "[]" > "$orphans_file"

  while IFS= read -r -d '' filepath; do
    local basename
    basename=$(basename "$filepath")
    [[ "$basename" == .* || "$basename" == "INDEX.jsonl" || "$basename" == "INDEX.md" ]] && continue

    local rel="${filepath#$CLAW_DRIVE_DIR/}"

    # Check if indexed (direct match or inside indexed directory)
    local found=false
    while IFS= read -r idx_path; do
      [[ -z "$idx_path" ]] && continue
      if [[ "$rel" == "$idx_path" ]]; then
        found=true; break
      fi
      if [[ -d "$CLAW_DRIVE_DIR/$idx_path" && "$rel" == "$idx_path"* ]]; then
        found=true; break
      fi
    done < "$indexed_paths_file"

    if [[ "$found" == "false" ]]; then
      local size_bytes modified
      size_bytes=$(stat -f%z "$filepath" 2>/dev/null || stat -c%s "$filepath" 2>/dev/null || echo 0)
      modified=$(stat -f%Sm -t "%Y-%m-%d" "$filepath" 2>/dev/null || stat -c%y "$filepath" 2>/dev/null | cut -d' ' -f1 || echo "unknown")

      local tmp_orphans
      tmp_orphans=$(safe_mktemp) || return 1
      jq -c --arg path "$rel" --argjson size "$size_bytes" --arg modified "$modified" \
        '. + [{path: $path, size_bytes: $size, modified: $modified, desc: "", tags: [], source: "", skip: false}]' \
        "$orphans_file" > "$tmp_orphans"
      mv "$tmp_orphans" "$orphans_file"
    fi
  done < <(find "$CLAW_DRIVE_DIR" -type f -not -path '*/.git/*' -not -name '.*' -not -name 'INDEX.jsonl' -not -name 'INDEX.md' -print0 | sort -z)

  # Build existing array with file_exists and size
  local existing_file
  existing_file=$(safe_mktemp) || return 1
  echo "[]" > "$existing_file"

  while IFS= read -r line; do
    [[ -z "$line" ]] && continue
    local path
    path=$(printf '%s' "$line" | jq -r '.path')
    local full="$CLAW_DRIVE_DIR/$path"
    local file_exists=false size_bytes=0

    if [[ -e "$full" ]]; then
      file_exists=true
      if [[ -f "$full" ]]; then
        size_bytes=$(stat -f%z "$full" 2>/dev/null || stat -c%s "$full" 2>/dev/null || echo 0)
      fi
    fi

    local tmp_existing
    tmp_existing=$(safe_mktemp) || return 1
    printf '%s' "$line" | jq -c --argjson exists "$file_exists" --argjson size "$size_bytes" \
      '. + {file_exists: $exists, size_bytes: $size}' | \
      jq -c --slurpfile arr "$existing_file" '$arr[0] + [.]' > "$tmp_existing"
    mv "$tmp_existing" "$existing_file"
  done < "$CLAW_DRIVE_INDEX"

  # Combine into plan
  jq -cn --slurpfile orphans "$orphans_file" --slurpfile existing "$existing_file" \
    '{orphans: $orphans[0], existing: $existing[0]}' > "$output"

  local orphan_count existing_count
  orphan_count=$(jq '.orphans | length' "$output")
  existing_count=$(jq '.existing | length' "$output")

  rm -f "$indexed_paths_file" "$orphans_file" "$existing_file"

  echo ""
  echo "📋 Plan written to: $output"
  echo "   Orphan files: $orphan_count"
  echo "   Existing entries: $existing_count"
  echo ""
  if [[ "$orphan_count" -gt 0 ]]; then
    echo "Orphans found:"
    jq -r '.orphans[] | "  📄 \(.path) (\(.size_bytes) bytes, \(.modified))"' "$output"
    echo ""
  fi
  echo "💡 Next: agent enriches the plan, then run 'claw-drive reindex apply $output'"
}

reindex_apply() {
  local plan="${1:-reindex-plan.json}"
  local dry_run=false
  shift || true

  while [[ $# -gt 0 ]]; do
    case "$1" in
      --dry-run) dry_run=true; shift ;;
      *) shift ;;
    esac
  done

  if [[ ! -f "$plan" ]]; then
    echo "❌ Plan file not found: $plan"
    return 1
  fi

  claw_drive_init || return 1

  echo "📥 Applying reindex plan: $plan"
  [[ "$dry_run" == "true" ]] && echo "   (dry run — no changes will be made)"
  echo ""

  local added=0 updated=0 skipped=0

  # Process orphans
  local orphan_count
  orphan_count=$(jq '.orphans | length' "$plan")

  local i=0
  while [[ $i -lt $orphan_count ]]; do
    local skip desc tags source path
    skip=$(jq -r ".orphans[$i].skip // false" "$plan")
    path=$(jq -r ".orphans[$i].path" "$plan")

    if [[ "$skip" == "true" ]]; then
      echo "  ⏭️  Skip: $path"
      ((skipped++)) || true
      ((i++)) || true
      continue
    fi

    desc=$(jq -r ".orphans[$i].desc // \"\"" "$plan")
    tags=$(jq -r ".orphans[$i].tags // [] | join(\",\")" "$plan")
    source=$(jq -r ".orphans[$i].source // \"reindex\"" "$plan")
    local metadata_json correspondent_val
    metadata_json=$(jq -c ".orphans[$i].metadata // null" "$plan")
    correspondent_val=$(jq -r ".orphans[$i].correspondent // \"\"" "$plan")

    if [[ -z "$desc" ]]; then
      echo "  ⚠️  No description for orphan: $path (skipping)"
      ((skipped++)) || true
      ((i++)) || true
      continue
    fi

    local date_str
    date_str=$(jq -r ".orphans[$i].modified // \"$(date +%Y-%m-%d)\"" "$plan")

    local meta_arg=""
    [[ "$metadata_json" != "null" ]] && meta_arg="$metadata_json"

    if [[ "$dry_run" == "true" ]]; then
      echo "  ➕ Would add: $path"
      echo "     desc: $desc"
      echo "     tags: $tags"
      [[ -n "$meta_arg" ]] && echo "     metadata: $meta_arg"
      [[ -n "$correspondent_val" ]] && echo "     correspondent: $correspondent_val"
    else
      # Add to index
      index_add "$date_str" "$path" "$desc" "$tags" "$source" "$meta_arg" "" "$correspondent_val"

      # Register hash
      local full="$CLAW_DRIVE_DIR/$path"
      if [[ -f "$full" ]]; then
        dedup_register "$full" "$path"
      fi

      echo "  ✅ Added: $path"
    fi
    ((added++)) || true
    ((i++)) || true
  done

  # Process existing entries with updates
  local existing_count
  existing_count=$(jq '.existing | length' "$plan")

  i=0
  while [[ $i -lt $existing_count ]]; do
    local new_desc new_tags new_metadata new_correspondent path
    path=$(jq -r ".existing[$i].path" "$plan")
    new_desc=$(jq -r ".existing[$i].new_desc // \"\"" "$plan")
    new_tags=$(jq -r ".existing[$i].new_tags // null" "$plan")
    new_metadata=$(jq -c ".existing[$i].new_metadata // null" "$plan")
    new_correspondent=$(jq -r ".existing[$i].new_correspondent // \"\"" "$plan")
    local new_source
    new_source=$(jq -r ".existing[$i].new_source // \"\"" "$plan")

    if [[ -n "$new_desc" || "$new_tags" != "null" || "$new_metadata" != "null" || -n "$new_correspondent" || -n "$new_source" ]]; then
      if [[ "$dry_run" == "true" ]]; then
        echo "  📝 Would update: $path"
        [[ -n "$new_desc" ]] && echo "     new desc: $new_desc"
        [[ "$new_tags" != "null" ]] && echo "     new tags: $(jq -r ".existing[$i].new_tags | join(\",\")" "$plan")"
        [[ "$new_metadata" != "null" ]] && echo "     new metadata: $new_metadata"
        [[ -n "$new_correspondent" ]] && echo "     new correspondent: $new_correspondent"
        [[ -n "$new_source" ]] && echo "     new source: $new_source"
      else
        local update_args=()
        [[ -n "$new_desc" ]] && update_args+=(--desc "$new_desc")
        if [[ "$new_tags" != "null" ]]; then
          local tags_csv
          tags_csv=$(jq -r ".existing[$i].new_tags | join(\",\")" "$plan")
          update_args+=(--tags "$tags_csv")
        fi
        [[ "$new_metadata" != "null" ]] && update_args+=(--metadata "$new_metadata")
        [[ -n "$new_correspondent" ]] && update_args+=(--correspondent "$new_correspondent")
        [[ -n "$new_source" ]] && update_args+=(--source "$new_source")

        index_update "$path" "${update_args[@]}"
        echo "  ✅ Updated: $path"
      fi
      ((updated++)) || true
    fi
    ((i++)) || true
  done

  echo ""
  echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
  if [[ "$dry_run" == "true" ]]; then
    echo "Dry run complete: $added would be added, $updated would be updated, $skipped skipped"
  else
    echo "✅ Reindex complete: $added added, $updated updated, $skipped skipped"
  fi
}


path: lib/sync.sh
content: #!/usr/bin/env bash
# lib/sync.sh — Google Drive sync daemon for Claw Drive

SYNC_DEBOUNCE_SEC=3
SYNC_AUTH_TIMEOUT=120

# Authenticate with Google Drive via rclone
sync_auth() {
  echo "🔐 Claw Drive — Google Drive Authorization"
  echo ""

  if ! command -v rclone &>/dev/null; then
    echo "❌ rclone not found. Install: brew install rclone"
    return 1
  fi

  if rclone listremotes 2>/dev/null | grep -q "^gdrive:$"; then
    echo "⚠️  rclone remote 'gdrive' already exists."
    echo "   To re-authorize, run: rclone config delete gdrive"
    echo "   Then run this command again."
    return 1
  fi

  echo "Starting rclone authorization..."
  echo ""

  # Start rclone in background so we can capture the token
  local rclone_out
  rclone_out=$(mktemp) || { echo "❌ mktemp failed"; return 1; }
  # Ensure token file is cleaned up on exit/interrupt
  trap 'rm -f "$rclone_out"' EXIT INT TERM

  rclone authorize "drive" --auth-no-open-browser > "$rclone_out" 2>&1 &
  local rclone_pid=$!

  # Wait for auth URL
  local waited=0
  local auth_url=""
  while [[ $waited -lt 10 ]]; do
    sleep 1
    ((waited++)) || true
    auth_url=$(grep -o 'http://127.0.0.1:53682/auth[?][^ ]*' "$rclone_out" 2>/dev/null | head -1 || true)
    [[ -n "$auth_url" ]] && break
  done

  if [[ -z "$auth_url" ]]; then
    echo "❌ rclone failed to start auth server."
    cat "$rclone_out"
    rm -f "$rclone_out"
    kill "$rclone_pid" 2>/dev/null
    return 1
  fi

  # Try to open browser on the machine
  open "$auth_url" 2>/dev/null || true

  echo "🌐 Auth URL: $auth_url"
  echo ""
  echo "⏳ Waiting for authorization..."
  echo "   Complete sign-in in the browser on this machine."

  # Wait for rclone to finish
  wait "$rclone_pid" 2>/dev/null
  local exit_code=$?

  if [[ $exit_code -ne 0 ]]; then
    echo "❌ Authorization failed or timed out."
    cat "$rclone_out"
    rm -f "$rclone_out"
    return 1
  fi

  # Extract token
  local token
  token=$(sed -n '/^{/,/^}/p' "$rclone_out" | head -20)
  rm -f "$rclone_out"

  if [[ -z "$token" ]]; then
    echo "❌ Could not extract token."
    return 1
  fi

  # Configure rclone remote
  rclone config create gdrive drive config_is_local=false config_token="$token" > /dev/null 2>&1

  echo "✅ Authorization successful!"
  echo "✅ rclone remote 'gdrive' configured."

  # Create default .sync-config
  if [[ ! -f "$CLAW_DRIVE_SYNC_CONFIG" ]]; then
    cat > "$CLAW_DRIVE_SYNC_CONFIG" <<EOF
backend: google-drive
remote: gdrive:claw-drive
exclude:
  - identity/
  - .hashes
EOF
    echo "✅ Created .sync-config"
  fi

  echo ""
  echo "🎉 Google Drive sync is ready!"
  echo "   Run: claw-drive sync start"
}

# Check sync prerequisites
sync_setup() {
  echo "🗄️  Claw Drive Sync Setup"
  echo ""

  local ok=true

  if command -v rclone &>/dev/null; then
    echo "✅ rclone installed ($(rclone version | head -1))"
  else
    echo "❌ rclone not found. Install: brew install rclone"
    ok=false
  fi

  if command -v fswatch &>/dev/null; then
    echo "✅ fswatch installed"
  else
    echo "❌ fswatch not found. Install: brew install fswatch"
    ok=false
  fi

  if [[ -d "$CLAW_DRIVE_DIR" ]]; then
    echo "✅ Drive directory: $CLAW_DRIVE_DIR"
  else
    echo "❌ Drive directory not found: $CLAW_DRIVE_DIR"
    ok=false
  fi

  local remote
  remote=$(sync_config_get "remote")
  if [[ -z "$remote" ]]; then
    echo ""
    echo "⚠️  No .sync-config found. Create $CLAW_DRIVE_SYNC_CONFIG:"
    echo ""
    echo "  backend: google-drive"
    echo "  remote: gdrive:claw-drive"
    echo "  exclude:"
    echo "    - identity/"
    echo "    - .hashes"
    return 1
  fi

  local remote_name="${remote%%:*}"
  if rclone listremotes | grep -q "^${remote_name}:$"; then
    echo "✅ rclone remote '$remote_name' configured"
  else
    echo "❌ rclone remote '$remote_name' not found. Run: rclone config"
    ok=false
  fi

  [[ "$ok" == "true" ]] || return 1

  echo ""
  echo "✅ Ready! Run: claw-drive sync start"
}

# One-shot sync
sync_push() {
  local remote
  remote=$(sync_config_get "remote")
  if [[ -z "$remote" ]]; then
    echo "❌ No remote configured. Run: claw-drive sync setup"
    return 1
  fi

  local exclude_args=()
  mapfile -t exclude_args < <(sync_build_exclude_args_lines)

  echo "📤 Syncing $CLAW_DRIVE_DIR → $remote ..."
  rclone sync "$CLAW_DRIVE_DIR" "$remote" "${exclude_args[@]}" --verbose 2>&1

  date -u +"%Y-%m-%dT%H:%M:%SZ" > "$CLAW_DRIVE_SYNC_STATE"
  echo "✅ Sync complete."
}

# Internal: fswatch loop (called by launchd)
sync_watch_loop() {
  local remote
  remote=$(sync_config_get "remote")
  if [[ -z "$remote" ]]; then
    echo "❌ No remote configured." >&2
    return 1
  fi

  local exclude_args=()
  mapfile -t exclude_args < <(sync_build_exclude_args_lines)

  echo "👀 Watching $CLAW_DRIVE_DIR for changes (debounce: ${SYNC_DEBOUNCE_SEC}s)..."
  echo "📡 Remote: $remote"

  fswatch -o -l "$SYNC_DEBOUNCE_SEC" \
    --exclude '\.sync-state$' \
    --exclude '\.sync-config$' \
    --exclude '\.DS_Store$' \
    "$CLAW_DRIVE_DIR" | while read -r _count; do
    echo "[$(date '+%H:%M:%S')] Change detected, syncing..."
    if rclone sync "$CLAW_DRIVE_DIR" "$remote" "${exclude_args[@]}" 2>&1; then
      date -u +"%Y-%m-%dT%H:%M:%SZ" > "$CLAW_DRIVE_SYNC_STATE"
      echo "[$(date '+%H:%M:%S')] ✅ Sync complete."
    else
      echo "[$(date '+%H:%M:%S')] ❌ Sync failed." >&2
    fi
  done
}

# Start the sync daemon via launchd
sync_start() {
  if launchctl list "$CLAW_DRIVE_PLIST_NAME" 2>/dev/null; then
    echo "⚠️  Already running. Use 'claw-drive sync stop' first."
    return 1
  fi

  mkdir -p "$CLAW_DRIVE_LOG_DIR"

  local script_path
  script_path="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/bin/claw-drive"

  cat > "$CLAW_DRIVE_PLIST_PATH" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>$CLAW_DRIVE_PLIST_NAME</string>
    <key>ProgramArguments</key>
    <array>
        <string>$script_path</string>
        <string>sync</string>
        <string>_watch</string>
    </array>
    <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
        <key>CLAW_DRIVE_DIR</key>
        <string>$CLAW_DRIVE_DIR</string>
    </dict>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>$CLAW_DRIVE_LOG_DIR/sync.log</string>
    <key>StandardErrorPath</key>
    <string>$CLAW_DRIVE_LOG_DIR/sync.err</string>
</dict>
</plist>
EOF

  launchctl load "$CLAW_DRIVE_PLIST_PATH"
  echo "✅ Sync daemon started."
  echo "   Logs: $CLAW_DRIVE_LOG_DIR/sync.log"
}

# Stop the sync daemon
sync_stop() {
  if [[ -f "$CLAW_DRIVE_PLIST_PATH" ]]; then
    launchctl unload "$CLAW_DRIVE_PLIST_PATH" 2>/dev/null || true
    rm -f "$CLAW_DRIVE_PLIST_PATH"
    echo "✅ Sync daemon stopped."
  else
    echo "⚠️  Not running."
  fi
}

# Show sync status
sync_status() {
  local format="${1:-table}"

  local running="false"
  if launchctl list "$CLAW_DRIVE_PLIST_NAME" &>/dev/null; then
    running="true"
  fi

  local remote
  remote=$(sync_config_get "remote")

  local last_sync="never"
  if [[ -f "$CLAW_DRIVE_SYNC_STATE" ]]; then
    last_sync=$(cat "$CLAW_DRIVE_SYNC_STATE")
  fi

  if [[ "$format" == "json" ]]; then
    printf '{"daemon_running":%s,"remote":"%s","last_sync":"%s"}\n' \
      "$running" "${remote:-null}" "$last_sync"
  else
    echo "🗄️  Claw Drive Sync Status"
    echo ""
    if [[ "$running" == "true" ]]; then
      echo "🟢 Daemon: running"
    else
      echo "🔴 Daemon: stopped"
    fi
    echo "📡 Remote: ${remote:-not configured}"
    echo "🕐 Last sync: $last_sync"

    local excludes
    excludes=$(sync_config_excludes)
    if [[ -n "$excludes" ]]; then
      echo "🚫 Excludes:"
      while IFS= read -r e; do
        echo "   - $e"
      done <<< "$excludes"
    fi
  fi
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)