Skill: billhandsome52/palest-ink (version 0.1.0)

--- name: palest-ink description: > Track and recall your daily activities including git commits, web browsing, shell commands, and VS Code edits. Use this skill whenever the user asks about their recent activity, wants a daily report or summary, asks "what did I do today/yesterday/this week", wants to find a specific commit or webpage they visited, asks about browsing history, needs to recall any past work activity, or queries about specific content they viewed online. Also trigger when the user mentions palest-ink, activity tracking, daily log, work journal, daily report, or activity summary. Trigger for questions like "which website had info about X", "when did I commit the code for Y", "show my git activity". tools: Read, Glob, Grep, Bash --- # Palest Ink (淡墨) — Activity Tracker & Daily Reporter > 好记性不如烂笔头 — The faintest ink is better than the strongest memory. ## Overview Palest Ink tracks the user's daily activities automatically: - **Git operations**: commits, pushes, pulls, branch switches - **Web browsing**: Chrome & Safari history with page content summaries - **Shell commands**: zsh/bash command history with execution duration - **VS Code edits**: recently opened/edited files - **App focus**: which application is in the foreground, with time duration - **File changes**: files modified in watched directories All data is stored locally at `~/.palest-ink/data/YYYY/MM/DD.jsonl`. ## Setup Check Before answering any query, first check if Palest Ink is installed: ```bash test -f ~/.palest-ink/config.json && echo "INSTALLED" || echo "NOT_INSTALLED" ``` If NOT installed, tell the user: > Palest Ink is not yet set up. To install, run: > ```bash > bash <SKILL_PATH>/../../collectors/install.sh > ``` > This will set up automatic tracking of git, browsing, and shell activity. Then stop and wait for the user to install. ## Answering Queries ### Daily Report / "What did I do today?" Run the report generator: ```bash python3 <SKILL_PATH>/scripts/report.py --date today ``` For yesterday: `--date yesterday` For a specific date: `--date 2026-03-03` For the whole week: `--week` Read the output and present it conversationally to the user. Highlight notable patterns (focused work sessions, frequent topics, etc). ### Searching for Specific Activities Use the query tool to search activity records: ```bash python3 <SKILL_PATH>/scripts/query.py --date today --type git_commit --search "plugin" ``` **Common query patterns:** | User asks about... | Arguments | |---------------------|-----------| | A git commit | `--type git_commit --search "keyword"` | | A webpage about X | `--type web_visit --search-content "keyword"` | | Shell commands | `--type shell_command --search "keyword"` | | VS Code files | `--type vscode_edit --search "keyword"` | | App focus / screen time | `--type app_focus --summary` | | File changes in project | `--type file_change --search "project"` | | Everything today | `--date today --summary` | | Date range | `--from 2026-03-01 --to 2026-03-07` | **Important:** When the user searches for web page content (e.g., "which website talked about homebrew"), use `--search-content` instead of `--search`. This searches within page content summaries and keywords, not just URLs and titles. ### Status Check Show collector status and data statistics: ```bash python3 <SKILL_PATH>/scripts/status.py ``` If the output contains "CLEANUP RECOMMENDED", proactively tell the user: > "Your palest-ink data is approaching 2 GB. Would you like me to clean up older records?" If the user agrees, first show a dry-run preview: ```bash python3 ~/.palest-ink/bin/cleanup.py --dry-run ``` Present the preview (how many files, date range, records count, space to free). Then ask for explicit confirmation before actually deleting: ```bash python3 ~/.palest-ink/bin/cleanup.py --force ``` Options: - `--max-size N` — threshold in GB (default: 2.0) - `--keep-days N` — always keep the most recent N days (default: 30) - `--dry-run` — preview only, no changes - `--force` — skip the interactive prompt (use after user confirms in chat) ## Fallback: Direct File Reading If scripts fail or for simple lookups, read the JSONL files directly: 1. Construct the file path: `~/.palest-ink/data/YYYY/MM/DD.jsonl` 2. Use Grep to search: `grep "keyword" ~/.palest-ink/data/2026/03/03.jsonl` 3. Each line is a JSON object with fields: `ts`, `type`, `source`, `data` ## Data Schema ### Activity Types - `git_commit` — data: repo, branch, hash, message, files_changed, insertions, deletions - `git_push` — data: repo, branch, remote, remote_url - `git_pull` — data: repo, branch, is_squash - `git_checkout` — data: repo, from_ref, to_branch - `web_visit` — data: url, title, visit_duration_seconds, browser, content_summary, content_keywords - `shell_command` — data: command, duration_seconds (null if not available) - `vscode_edit` — data: file_path, workspace, language - `app_focus` — data: app_name, window_title, duration_seconds - `file_change` — data: path, workspace, language, event ### Web Visit Content Web visits include a `content_summary` field (up to 800 chars of page text) and `content_keywords` (extracted keywords). This enables content-based search. Example: if user browsed a page about "Homebrew installation guide", the content_summary will contain the actual page text, making it searchable even if the URL/title don't mention it. ## Tips for Good Answers 1. When showing git activity, include the commit message and changed files 2. When showing web visits, include both the title and a brief content summary 3. For "what did I do" questions, give a narrative summary, not just raw data 4. Group related activities together (e.g., "You worked on project X, making 5 commits...") 5. If the search returns too many results, help the user narrow down 6. Mention the time of activities to give temporal context

path: README.md
content: # Palest Ink (淡墨)

> 好记性不如烂笔头 — The faintest ink is better than the strongest memory.

A Claude Code skill that automatically tracks your daily development activities and helps you recall what you've done — without lifting a finger.

## Features

### Data Collection (automatic, every 15 seconds via launchd)

| Collector | What it tracks |
|-----------|---------------|
| **Git** | Commits, pushes, pulls, branch switches (via global git hooks + periodic scan) |
| **Browser** | Chrome & Safari history with full page content summaries |
| **Shell** | zsh/bash command history with execution duration |
| **VS Code** | Recently opened/edited files |
| **App Focus** | Which application is in the foreground and how long |
| **File Changes** | Files modified in your watched project directories |

### Reports

Ask Claude *"What did I do today?"* and get a structured daily report including:

- **Timeline** — chronological activity across morning / afternoon / evening / night
- **Git Activity** — commits per repo with line change stats
- **Top Websites** — domains visited with visit counts
- **Files Edited** — VS Code edits grouped by language
- **专注时段 (Focus Sessions)** — continuous blocks of focus on a single app (e.g. "09:00–11:15 Cursor 2h15m")
- **应用使用时长 (App Usage)** — Top 5 apps by time + breakdown by category (development / browser / communication)
- **Shell 命令统计 (Shell Stats)** — most-used commands + slowest commands by duration
- **跨域关联 (Cross-domain Correlation)** — web research topics detected in the 2 hours before each git commit

### Natural Language Search

| You ask | How it's answered |
|---------|-------------------|
| "Which commit touched the auth code?" | Searches `git_commit` records by keyword |
| "Which website had info about Homebrew?" | Searches **page content summaries**, not just titles/URLs |
| "How long did I use Cursor today?" | Queries `app_focus` records and sums duration |
| "What shell commands took more than 30 seconds?" | Queries `shell_command` with `duration_seconds` |
| "What files changed in the palest-ink project?" | Queries `file_change` records |
| "Show my activity from last Monday to Wednesday" | Date-range query across multiple days |

## Installation

### 1. Install the collectors

```bash
bash collectors/install.sh
```

This will:
- Create `~/.palest-ink/` for storing activity data
- Write a default `config.json`
- Install git hooks globally (`post-commit`, `post-merge`, `post-checkout`, `pre-push`)
- Install a **launchd agent** that runs every 15 seconds (replaces cron)

### 2. Grant permissions

**Full Disk Access** (for Safari history):
> System Settings → Privacy & Security → Full Disk Access → enable Terminal.app

**Accessibility** (for app focus tracking):
> System Settings → Privacy & Security → Accessibility → enable Terminal.app

### 3. Install as a Claude Code skill

Add the skill to your Claude Code settings by pointing to the `skills/` directory, or follow Claude Code's plugin installation instructions.

### 4. (Optional) Configure watched directories

Edit `~/.palest-ink/config.json` to set which directories to monitor for file changes:

```json
"watched_dirs": ["/Users/you/projects/myapp", "/Users/you/work"]
```

If `watched_dirs` is empty, the file change collector falls back to `tracked_repos`.

## Usage Examples

Once installed, just talk to Claude:

```
"今天做了什么？"
"What did I do today?"
"Show my git activity this week"
"Which website had information about JWT authentication?"
"How long was I in Cursor vs Chrome today?"
"What files did I change in the palest-ink project today?"
"Which shell commands took the longest to run this week?"
"Show me what I researched before my last commit"
```

## Data Storage

All data stays local on your machine — nothing leaves your machine.

```
~/.palest-ink/
├── config.json            # Configuration and collector state
├── data/
│   └── YYYY/MM/DD.jsonl   # Activity records (one JSON object per line)
├── reports/               # Saved report files
├── hooks/                 # Git hook scripts
├── bin/                   # Collector scripts (copied from collectors/)
├── tmp/                   # Lock files, marker files, temp files
└── cron.log               # Collection log
```

Each record follows a common schema:

```json
{
  "ts": "2026-03-03T09:14:22+00:00",
  "type": "git_commit",
  "source": "git_hook",
  "data": { ... }
}
```

Supported types: `git_commit`, `git_push`, `git_pull`, `git_checkout`, `web_visit`, `shell_command`, `vscode_edit`, `app_focus`, `file_change`.

See [`skills/palest-ink/references/schema.md`](skills/palest-ink/references/schema.md) for full field definitions.

## Configuration

`~/.palest-ink/config.json` — key fields:

```json
{
  "collectors": {
    "chrome": true,
    "safari": true,
    "shell": true,
    "vscode": true,
    "git_hooks": true,
    "git_scan": true,
    "content": true,
    "app": true,
    "fsevent": true
  },
  "tracked_repos": ["/path/to/repo"],
  "watched_dirs": [],
  "exclude_patterns": {
    "urls": ["chrome://", "chrome-extension://", "about:", "file://"],
    "commands": ["^ls$", "^cd ", "^clear$", "^pwd$", "^exit$", "^history"]
  },
  "content_fetch": {
    "max_urls_per_run": 50,
    "summary_max_chars": 800,
    "timeout_seconds": 10
  },
  "app": {
    "min_focus_seconds": 600,
    "exclude": ["loginwindow", "Dock", "SystemUIServer", "Finder", "ScreenSaverEngine"]
  },
  "app_categories": {
    "development": ["Cursor", "Code", "Xcode", "Terminal", "iTerm2", "Warp"],
    "browser": ["Google Chrome", "Safari", "Firefox", "Arc"],
    "communication": ["Slack", "WeChat", "Discord", "Zoom"]
  }
}
```

## Uninstallation

```bash
bash collectors/uninstall.sh
```

Removes the launchd agent, restores git hooks, and optionally deletes collected data.

## Privacy

- All data is stored **locally** — nothing is sent anywhere
- URL exclusion patterns filter out browser-internal and sensitive pages
- Command exclusion patterns filter out noise (`ls`, `cd`, `clear`, etc.)
- App focus collection skips the lock screen (`loginwindow`, `ScreenSaverEngine`)
- Accessible only to your local user account

## License

MIT


path: collectors/cleanup.py
content: #!/usr/bin/env python3
"""Palest Ink - Data Cleanup Tool

Removes oldest activity records when storage exceeds a threshold.
Always asks for user confirmation before deleting (unless --force).

Usage:
    python3 cleanup.py                   # Check size; clean if > 2 GB
    python3 cleanup.py --dry-run         # Show what would be deleted, no changes
    python3 cleanup.py --max-size 1.5    # Custom threshold in GB
    python3 cleanup.py --keep-days 60    # Always keep the most recent N days
    python3 cleanup.py --force           # Skip confirmation prompt (for skill use)
"""

import argparse
import json
import os
import sys
from datetime import datetime, timedelta, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")
FLAG_FILE = os.path.join(PALEST_INK_DIR, "tmp", "cleanup_needed")

DEFAULT_MAX_SIZE_GB = 2.0
DEFAULT_KEEP_DAYS = 30
TARGET_RATIO = 0.70  # After cleanup, aim for 70% of max_size


def format_size(size_bytes):
    for unit in ["B", "KB", "MB", "GB"]:
        if size_bytes < 1024:
            return f"{size_bytes:.1f} {unit}"
        size_bytes /= 1024
    return f"{size_bytes:.1f} TB"


def get_all_datafiles():
    """Return list of (date_str, filepath, size_bytes) sorted oldest-first."""
    files = []
    if not os.path.isdir(DATA_DIR):
        return files
    for year in sorted(os.listdir(DATA_DIR)):
        year_dir = os.path.join(DATA_DIR, year)
        if not os.path.isdir(year_dir) or not year.isdigit():
            continue
        for month in sorted(os.listdir(year_dir)):
            month_dir = os.path.join(year_dir, month)
            if not os.path.isdir(month_dir) or not month.isdigit():
                continue
            for fname in sorted(os.listdir(month_dir)):
                if not fname.endswith(".jsonl"):
                    continue
                day = fname.replace(".jsonl", "")
                date_str = f"{year}-{month}-{day}"
                fpath = os.path.join(month_dir, fname)
                size = os.path.getsize(fpath)
                files.append((date_str, fpath, size))
    return files


def count_records(filepath):
    try:
        with open(filepath, "r", errors="replace") as f:
            return sum(1 for line in f if line.strip())
    except OSError:
        return 0


def remove_empty_dirs(path):
    """Recursively remove empty directories up to DATA_DIR."""
    while path != DATA_DIR:
        try:
            if not os.listdir(path):
                os.rmdir(path)
            else:
                break
        except OSError:
            break
        path = os.path.dirname(path)


def clear_flag():
    try:
        os.unlink(FLAG_FILE)
    except OSError:
        pass


def main():
    parser = argparse.ArgumentParser(description="Palest Ink - Data Cleanup Tool")
    parser.add_argument("--max-size", type=float, default=DEFAULT_MAX_SIZE_GB,
                        metavar="GB", help=f"Size threshold in GB (default: {DEFAULT_MAX_SIZE_GB})")
    parser.add_argument("--keep-days", type=int, default=DEFAULT_KEEP_DAYS,
                        metavar="N", help=f"Always keep last N days (default: {DEFAULT_KEEP_DAYS})")
    parser.add_argument("--dry-run", action="store_true",
                        help="Show what would be deleted without making changes")
    parser.add_argument("--force", action="store_true",
                        help="Skip confirmation prompt")
    args = parser.parse_args()

    max_bytes = int(args.max_size * 1024 ** 3)
    target_bytes = int(max_bytes * TARGET_RATIO)
    keep_cutoff = (datetime.now() - timedelta(days=args.keep_days)).strftime("%Y-%m-%d")

    all_files = get_all_datafiles()
    if not all_files:
        print("No data files found.")
        return

    total_bytes = sum(s for _, _, s in all_files)
    total_records = None  # compute lazily only if needed

    print(f"Data directory : {DATA_DIR}")
    print(f"Current size   : {format_size(total_bytes)}")
    print(f"Threshold      : {format_size(max_bytes)}")
    print(f"Date range     : {all_files[0][0]}  →  {all_files[-1][0]}")
    print(f"Total files    : {len(all_files)}")
    print()

    if total_bytes <= max_bytes:
        print(f"Storage is within the {args.max_size} GB limit. No cleanup needed.")
        clear_flag()
        return

    overage = total_bytes - max_bytes
    print(f"⚠️  Exceeds limit by {format_size(overage)}. "
          f"Will delete oldest files until size ≤ {format_size(target_bytes)} "
          f"({int(TARGET_RATIO * 100)}% of threshold).")
    print()

    # Determine which files to delete (oldest first, never touch keep_cutoff window)
    to_delete = []
    freed = 0
    remaining = total_bytes

    for date_str, fpath, size in all_files:
        if remaining <= target_bytes:
            break
        if date_str >= keep_cutoff:
            # Reached the protected window — stop
            break
        to_delete.append((date_str, fpath, size))
        freed += size
        remaining -= size

    if not to_delete:
        print(f"Cannot free enough space without deleting files from the last "
              f"{args.keep_days} days (protected window: {keep_cutoff} → today).")
        print("Use --keep-days to reduce the protected window, or manually delete files.")
        return

    # Collect stats
    record_count = sum(count_records(fp) for _, fp, _ in to_delete)
    date_from = to_delete[0][0]
    date_to = to_delete[-1][0]

    print(f"Files to delete : {len(to_delete)}  ({date_from}  →  {date_to})")
    print(f"Records removed : {record_count:,}")
    print(f"Space freed     : {format_size(freed)}")
    print(f"Size after      : {format_size(remaining)}")
    print()

    if args.dry_run:
        print("Dry run — no files were deleted.")
        print("\nFiles that would be removed:")
        for date_str, fpath, size in to_delete:
            nrec = count_records(fpath)
            print(f"  {date_str}  {format_size(size):>10}  ({nrec} records)")
        return

    # Confirmation
    if not args.force:
        answer = input(
            f"Delete {len(to_delete)} files ({format_size(freed)}) "
            f"from {date_from} to {date_to}? [y/N] "
        ).strip().lower()
        if answer not in ("y", "yes"):
            print("Aborted. No files were deleted.")
            return

    # Delete
    deleted_count = 0
    deleted_bytes = 0
    for date_str, fpath, size in to_delete:
        try:
            os.unlink(fpath)
            remove_empty_dirs(os.path.dirname(fpath))
            deleted_count += 1
            deleted_bytes += size
        except OSError as e:
            print(f"  Warning: could not delete {fpath}: {e}")

    clear_flag()
    print(f"\nDone. Deleted {deleted_count} files, freed {format_size(deleted_bytes)}.")
    print(f"Current data size: {format_size(remaining)}")


if __name__ == "__main__":
    main()


path: collectors/collect_app.py
content: #!/usr/bin/env python3
"""Palest Ink - App Focus Collector

Tracks which application is in the foreground using osascript.
Merges duration in-place when the same app+window persists across runs.
"""

import json
import os
import subprocess
import tempfile
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")

LOCK_APPS = {"loginwindow", "ScreenSaverEngine"}


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def get_frontmost_app():
    """Return (app_name, window_title) of the frontmost application, or (None, None) on failure."""
    script = '''
tell application "System Events"
    set frontApp to name of first application process whose frontmost is true
end tell
tell application frontApp
    try
        set winTitle to name of front window
    on error
        set winTitle to ""
    end try
end tell
return frontApp & "|||" & winTitle
'''
    try:
        result = subprocess.run(
            ["osascript", "-e", script],
            capture_output=True, text=True, timeout=3
        )
        if result.returncode != 0:
            return None, None
        output = result.stdout.strip()
        if "|||" in output:
            parts = output.split("|||", 1)
            return parts[0].strip(), parts[1].strip()
        return output, ""
    except (subprocess.TimeoutExpired, OSError):
        return None, None


def atomic_update_line(filepath, line_number, new_record):
    """Replace a specific line in a file atomically using a temp file."""
    try:
        with open(filepath, "r") as f:
            lines = f.readlines()
        if line_number < 1 or line_number > len(lines):
            return False
        lines[line_number - 1] = json.dumps(new_record, ensure_ascii=False) + "\n"
        tmp_fd, tmp_path = tempfile.mkstemp(dir=os.path.dirname(filepath))
        try:
            with os.fdopen(tmp_fd, "w") as tmp_f:
                tmp_f.writelines(lines)
            os.replace(tmp_path, filepath)
        except Exception:
            try:
                os.unlink(tmp_path)
            except OSError:
                pass
            return False
        return True
    except (OSError, IOError):
        return False


def count_lines(filepath):
    """Count lines in a file."""
    try:
        with open(filepath, "r") as f:
            return sum(1 for _ in f)
    except (OSError, IOError):
        return 0


def collect():
    config = load_config()
    if not config.get("collectors", {}).get("app", True):
        return

    app_exclude = set(config.get("app", {}).get("exclude", [
        "loginwindow", "Dock", "SystemUIServer", "Finder", "ScreenSaverEngine"
    ]))

    app_name, window_title = get_frontmost_app()
    if app_name is None:
        return

    # Skip lock screen / screen saver
    if app_name in LOCK_APPS or app_name in app_exclude:
        return

    now = datetime.now(timezone.utc)
    now_str = now.isoformat()
    today_str = now.strftime("%Y-%m-%d")

    # Check if same app+window as last time
    last_app = config.get("app_last_app", "")
    last_window = config.get("app_last_window", "")
    last_ts_str = config.get("app_last_ts", "")
    last_line = config.get("app_last_record_line", -1)

    same_app = (app_name == last_app and window_title == last_window)

    if same_app and last_ts_str and last_line >= 1:
        # Determine if last record is from same day
        try:
            last_dt = datetime.fromisoformat(last_ts_str)
            if last_dt.tzinfo is None:
                last_dt = last_dt.replace(tzinfo=timezone.utc)
            last_day = last_dt.astimezone(timezone.utc).strftime("%Y-%m-%d")
        except Exception:
            last_day = ""

        if last_day == today_str:
            # Try to merge duration into last record
            datafile = get_datafile(now)
            # Duration = time since last record ts
            try:
                elapsed = int((now - last_dt).total_seconds())
            except Exception:
                elapsed = 15  # fallback

            # Read existing record
            try:
                with open(datafile, "r") as f:
                    lines = f.readlines()
                if 1 <= last_line <= len(lines):
                    existing = json.loads(lines[last_line - 1])
                    prev_duration = existing.get("data", {}).get("duration_seconds", 0)
                    existing["data"]["duration_seconds"] = prev_duration + elapsed
                    if atomic_update_line(datafile, last_line, existing):
                        config["app_last_ts"] = now_str
                        save_config(config)
                        return
            except (OSError, json.JSONDecodeError):
                pass
            # Fallback: write new record below

    # Write new record
    datafile = get_datafile(now)
    record = {
        "ts": now_str,
        "type": "app_focus",
        "source": "app_collector",
        "data": {
            "app_name": app_name,
            "window_title": window_title,
            "duration_seconds": 15,
        }
    }

    with open(datafile, "a") as f:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

    new_line = count_lines(datafile)
    config["app_last_app"] = app_name
    config["app_last_window"] = window_title
    config["app_last_ts"] = now_str
    config["app_last_record_line"] = new_line
    save_config(config)

    print(f"[app] {app_name} — {window_title[:60]}")


if __name__ == "__main__":
    collect()


path: collectors/collect_chrome.py
content: #!/usr/bin/env python3
"""Palest Ink - Chrome History Collector

Reads Chrome's SQLite History database and extracts browsing records.
Uses a high-water mark to avoid re-processing visited URLs.
"""

import json
import os
import shutil
import sqlite3
import tempfile
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")
CHROME_HISTORY = os.path.expanduser(
    "~/Library/Application Support/Google/Chrome/Default/History"
)

# Chrome timestamps are microseconds since 1601-01-01 (WebKit epoch)
WEBKIT_EPOCH_OFFSET = 11644473600


def chrome_time_to_datetime(chrome_ts):
    """Convert Chrome/WebKit timestamp to Python datetime."""
    if not chrome_ts or chrome_ts <= 0:
        return None
    unix_ts = (chrome_ts / 1_000_000) - WEBKIT_EPOCH_OFFSET
    try:
        return datetime.fromtimestamp(unix_ts, tz=timezone.utc)
    except (OSError, ValueError):
        return None


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    """Get the JSONL data file path for a given datetime."""
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def should_exclude(url, exclude_patterns):
    """Check if URL matches any exclusion pattern."""
    for pattern in exclude_patterns:
        if url.startswith(pattern):
            return True
    return False


def collect():
    if not os.path.exists(CHROME_HISTORY):
        return

    config = load_config()
    if not config.get("collectors", {}).get("chrome", True):
        return

    last_visit_id = config.get("chrome_last_visit_id", 0)
    exclude_patterns = config.get("exclude_patterns", {}).get("urls", [])

    # Copy DB to avoid Chrome's lock
    tmp_fd, tmp_path = tempfile.mkstemp(suffix=".db")
    os.close(tmp_fd)
    try:
        shutil.copy2(CHROME_HISTORY, tmp_path)
        conn = sqlite3.connect(tmp_path)
        conn.row_factory = sqlite3.Row

        rows = conn.execute("""
            SELECT v.id, u.url, u.title, v.visit_time, v.visit_duration
            FROM visits v
            JOIN urls u ON v.url = u.id
            WHERE v.id > ?
            ORDER BY v.id
        """, (last_visit_id,)).fetchall()

        new_last_id = last_visit_id
        records_by_file = {}

        for row in rows:
            vid = row["id"]
            url = row["url"] or ""
            title = row["title"] or ""
            vtime = row["visit_time"]
            vduration = row["visit_duration"]

            if should_exclude(url, exclude_patterns):
                new_last_id = max(new_last_id, vid)
                continue

            dt = chrome_time_to_datetime(vtime)
            if dt is None:
                new_last_id = max(new_last_id, vid)
                continue

            duration_sec = (vduration // 1_000_000) if vduration and vduration > 0 else 0

            record = {
                "ts": dt.isoformat(),
                "type": "web_visit",
                "source": "chrome_collector",
                "data": {
                    "url": url,
                    "title": title,
                    "visit_duration_seconds": duration_sec,
                    "browser": "chrome",
                    "content_pending": True
                }
            }

            datafile = get_datafile(dt)
            if datafile not in records_by_file:
                records_by_file[datafile] = []
            records_by_file[datafile].append(record)
            new_last_id = max(new_last_id, vid)

        conn.close()

        # Write records grouped by date file
        for datafile, records in records_by_file.items():
            with open(datafile, "a") as f:
                for record in records:
                    f.write(json.dumps(record, ensure_ascii=False) + "\n")

        # Update high-water mark
        config["chrome_last_visit_id"] = new_last_id
        save_config(config)

        if rows:
            print(f"[chrome] Collected {len(rows)} visits (last_id: {new_last_id})")

    finally:
        os.unlink(tmp_path)


if __name__ == "__main__":
    collect()


path: collectors/collect_content.py
content: #!/usr/bin/env python3
"""Palest Ink - Web Page Content Collector

Fetches web page content for URLs marked with content_pending=true.
Extracts main text content and keywords, then updates the JSONL records.
"""

import json
import os
import re
import tempfile
from datetime import datetime, timezone
from html.parser import HTMLParser
from urllib.request import urlopen, Request
from urllib.error import URLError, HTTPError

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")

# Tags whose content should be ignored
IGNORE_TAGS = {"script", "style", "noscript", "header", "nav", "footer", "aside", "iframe"}

# Common stop words for keyword extraction (English + Chinese punctuation)
STOP_WORDS = {
    "the", "a", "an", "is", "are", "was", "were", "be", "been", "being",
    "have", "has", "had", "do", "does", "did", "will", "would", "shall",
    "should", "may", "might", "must", "can", "could", "and", "but", "or",
    "nor", "not", "so", "yet", "for", "of", "in", "on", "at", "to", "from",
    "by", "with", "as", "this", "that", "these", "those", "it", "its",
    "i", "you", "he", "she", "we", "they", "me", "him", "her", "us", "them",
    "my", "your", "his", "our", "their", "what", "which", "who", "whom",
    "if", "then", "than", "when", "where", "how", "all", "each", "every",
    "both", "few", "more", "most", "other", "some", "such", "no", "only",
    "own", "same", "too", "very", "just", "about", "above", "after", "before",
    "between", "into", "through", "during", "up", "down", "out", "off", "over",
    "under", "again", "further", "once", "here", "there", "any", "also",
    "的", "了", "在", "是", "我", "有", "和", "就", "不", "人", "都", "一",
    "这", "中", "大", "为", "上", "个", "国", "他", "到", "说", "们", "会",
}


class TextExtractor(HTMLParser):
    """Extract readable text from HTML, ignoring scripts/styles/nav."""

    def __init__(self):
        super().__init__()
        self.text_parts = []
        self.ignore_depth = 0
        self.title = ""
        self._in_title = False

    def handle_starttag(self, tag, attrs):
        tag = tag.lower()
        if tag in IGNORE_TAGS:
            self.ignore_depth += 1
        if tag == "title":
            self._in_title = True

    def handle_endtag(self, tag):
        tag = tag.lower()
        if tag in IGNORE_TAGS and self.ignore_depth > 0:
            self.ignore_depth -= 1
        if tag == "title":
            self._in_title = False
        # Add space after block elements
        if tag in ("p", "div", "h1", "h2", "h3", "h4", "h5", "h6",
                    "li", "td", "th", "br", "tr", "section", "article"):
            self.text_parts.append(" ")

    def handle_data(self, data):
        if self._in_title:
            self.title += data
        if self.ignore_depth == 0:
            self.text_parts.append(data)

    def get_text(self):
        text = " ".join(self.text_parts)
        # Collapse whitespace
        text = re.sub(r'\s+', ' ', text).strip()
        return text


def extract_keywords(text, max_keywords=10):
    """Extract keywords from text using simple word frequency."""
    # Split into words (handles both English and Chinese via simple approach)
    words = re.findall(r'[\w\u4e00-\u9fff]{2,}', text.lower())
    freq = {}
    for w in words:
        if w not in STOP_WORDS and not w.isdigit():
            freq[w] = freq.get(w, 0) + 1

    sorted_words = sorted(freq.items(), key=lambda x: x[1], reverse=True)
    return [w for w, _ in sorted_words[:max_keywords]]


def fetch_content(url, timeout=10):
    """Fetch and extract text content from a URL."""
    try:
        req = Request(url, headers={
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
        })
        with urlopen(req, timeout=timeout) as resp:
            content_type = resp.headers.get("Content-Type", "")
            if "text/html" not in content_type and "text/plain" not in content_type:
                return None, None

            charset = "utf-8"
            if "charset=" in content_type:
                charset = content_type.split("charset=")[-1].split(";")[0].strip()

            html = resp.read(500_000).decode(charset, errors="replace")

        parser = TextExtractor()
        parser.feed(html)
        text = parser.get_text()
        return text, parser.title

    except (URLError, HTTPError, OSError, UnicodeDecodeError, ValueError) as e:
        return None, None


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def collect():
    config = load_config()
    if not config.get("collectors", {}).get("content", True):
        return

    content_cfg = config.get("content_fetch", {})
    max_urls = content_cfg.get("max_urls_per_run", 50)
    summary_max = content_cfg.get("summary_max_chars", 800)
    timeout = content_cfg.get("timeout_seconds", 10)

    # Find today's and yesterday's data files
    now = datetime.now(timezone.utc)
    dates_to_check = [now]
    # Also check yesterday (for late-night browsing)
    from datetime import timedelta
    dates_to_check.append(now - timedelta(days=1))

    urls_processed = 0

    for dt in dates_to_check:
        datafile = os.path.join(
            DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl"
        )
        if not os.path.exists(datafile):
            continue

        # Read all records
        with open(datafile, "r") as f:
            lines = f.readlines()

        updated = False
        new_lines = []
        seen_urls = set()

        for line in lines:
            line = line.strip()
            if not line:
                new_lines.append(line)
                continue

            try:
                record = json.loads(line)
            except json.JSONDecodeError:
                new_lines.append(line)
                continue

            if (record.get("type") == "web_visit"
                    and record.get("data", {}).get("content_pending") is True
                    and urls_processed < max_urls):

                url = record["data"].get("url", "")

                # Skip duplicate URLs in same file
                if url in seen_urls:
                    record["data"]["content_pending"] = False
                    record["data"]["content_note"] = "duplicate_url"
                    new_lines.append(json.dumps(record, ensure_ascii=False))
                    updated = True
                    continue

                seen_urls.add(url)
                text, page_title = fetch_content(url, timeout=timeout)

                if text:
                    record["data"]["content_summary"] = text[:summary_max]
                    record["data"]["content_keywords"] = extract_keywords(text)
                    if page_title and not record["data"].get("title"):
                        record["data"]["title"] = page_title.strip()
                else:
                    record["data"]["content_error"] = True

                record["data"]["content_pending"] = False
                urls_processed += 1
                updated = True
                new_lines.append(json.dumps(record, ensure_ascii=False))
            else:
                new_lines.append(line)

        if updated:
            # Atomic write: write to temp file then rename
            tmp_fd, tmp_path = tempfile.mkstemp(
                dir=os.path.dirname(datafile), suffix=".tmp"
            )
            with os.fdopen(tmp_fd, "w") as f:
                f.write("\n".join(new_lines) + "\n")
            os.replace(tmp_path, datafile)

    if urls_processed > 0:
        print(f"[content] Fetched content for {urls_processed} URLs")


if __name__ == "__main__":
    collect()


path: collectors/collect_fsevent.py
content: #!/usr/bin/env python3
"""Palest Ink - Filesystem Change Collector

Detects recently modified files using a marker file + find -newer.
Monitors watched_dirs from config, or falls back to tracked_repos.
"""

import json
import os
import subprocess
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")
MARKER_FILE = os.path.join(PALEST_INK_DIR, "tmp", "fsevent_marker")


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def guess_language(filepath):
    """Guess programming language from file extension (shared logic with collect_vscode.py)."""
    ext_map = {
        ".py": "python", ".js": "javascript", ".ts": "typescript",
        ".tsx": "typescript-react", ".jsx": "javascript-react",
        ".java": "java", ".go": "go", ".rs": "rust", ".rb": "ruby",
        ".php": "php", ".c": "c", ".cpp": "cpp", ".h": "c-header",
        ".cs": "csharp", ".swift": "swift", ".kt": "kotlin",
        ".scala": "scala", ".r": "r", ".R": "r",
        ".sql": "sql", ".sh": "shell", ".bash": "shell", ".zsh": "shell",
        ".html": "html", ".css": "css", ".scss": "scss", ".less": "less",
        ".json": "json", ".yaml": "yaml", ".yml": "yaml", ".toml": "toml",
        ".xml": "xml", ".md": "markdown", ".rst": "restructuredtext",
        ".vue": "vue", ".svelte": "svelte",
    }
    _, ext = os.path.splitext(filepath)
    return ext_map.get(ext.lower(), "unknown")


def find_workspace(filepath):
    """Walk up directory tree to find nearest .git directory."""
    directory = os.path.dirname(os.path.abspath(filepath))
    while directory != os.path.dirname(directory):  # stop at filesystem root
        if os.path.isdir(os.path.join(directory, ".git")):
            return directory
        directory = os.path.dirname(directory)
    return ""


def find_changed_files(watch_dir, marker_path):
    """Use find -newer to get files changed since marker mtime."""
    try:
        result = subprocess.run(
            [
                "find", watch_dir,
                "-newer", marker_path,
                "-maxdepth", "4",
                "-type", "f",
                "!", "-path", "*/.git/*",
                "!", "-path", "*/node_modules/*",
                "!", "-path", "*/__pycache__/*",
                "!", "-name", ".DS_Store",
            ],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode != 0:
            return []
        return [p for p in result.stdout.strip().split("\n") if p]
    except (subprocess.TimeoutExpired, OSError):
        return []


def collect():
    config = load_config()
    if not config.get("collectors", {}).get("fsevent", True):
        return

    # Determine watched directories
    watched_dirs = config.get("watched_dirs", [])
    if not watched_dirs:
        watched_dirs = config.get("tracked_repos", [])

    if not watched_dirs:
        return

    # First run: create marker and exit
    first_run = not os.path.exists(MARKER_FILE)
    if first_run:
        os.makedirs(os.path.dirname(MARKER_FILE), exist_ok=True)
        open(MARKER_FILE, "w").close()
        return

    now = datetime.now(timezone.utc)
    changed_files = []
    for watch_dir in watched_dirs:
        if os.path.isdir(watch_dir):
            changed_files.extend(find_changed_files(watch_dir, MARKER_FILE))

    # Update marker timestamp before writing records
    os.utime(MARKER_FILE, None)

    if not changed_files:
        return

    records_by_file = {}
    count = 0
    seen_paths = set()

    for filepath in changed_files:
        filepath = os.path.normpath(filepath)
        if filepath in seen_paths:
            continue
        seen_paths.add(filepath)

        # Skip binary-like files and hidden system files
        filename = os.path.basename(filepath)
        if filename.startswith(".") and filename not in {".env", ".gitignore", ".eslintrc"}:
            continue

        workspace = find_workspace(filepath)
        language = guess_language(filepath)

        record = {
            "ts": now.isoformat(),
            "type": "file_change",
            "source": "fsevent_collector",
            "data": {
                "path": filepath,
                "workspace": workspace,
                "language": language,
                "event": "modified",
            }
        }

        datafile = get_datafile(now)
        if datafile not in records_by_file:
            records_by_file[datafile] = []
        records_by_file[datafile].append(record)
        count += 1

    for datafile, records in records_by_file.items():
        with open(datafile, "a") as f:
            for record in records:
                f.write(json.dumps(record, ensure_ascii=False) + "\n")

    if count > 0:
        print(f"[fsevent] Recorded {count} file changes")


if __name__ == "__main__":
    collect()


path: collectors/collect_git.py
content: #!/usr/bin/env python3
"""Palest Ink - Git Repository Scanner

Scans tracked git repositories for recent commits by the current user.
Supplements git hooks by catching commits made without hooks (IDE, etc).
"""

import json
import os
import subprocess
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def get_existing_hashes(datafile):
    """Read existing commit hashes from today's data to avoid duplicates."""
    hashes = set()
    if not os.path.exists(datafile):
        return hashes
    with open(datafile, "r") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                record = json.loads(line)
                if record.get("type") == "git_commit":
                    h = record.get("data", {}).get("hash", "")
                    if h:
                        hashes.add(h)
            except json.JSONDecodeError:
                pass
    return hashes


def run_git(repo_path, args):
    """Run a git command in the specified repo."""
    try:
        result = subprocess.run(
            ["git"] + args,
            cwd=repo_path,
            capture_output=True,
            text=True,
            timeout=10
        )
        return result.stdout.strip() if result.returncode == 0 else ""
    except (subprocess.TimeoutExpired, OSError):
        return ""


def scan_repo(repo_path, since_ts, existing_hashes):
    """Scan a single repo for recent commits."""
    if not os.path.isdir(os.path.join(repo_path, ".git")):
        return []

    # Get current user's email
    user_email = run_git(repo_path, ["config", "user.email"])
    if not user_email:
        return []

    since_arg = since_ts if since_ts else "1 day ago"

    # Get recent commits by this user
    log_format = "%H|||%s|||%aI|||%D"
    log_output = run_git(repo_path, [
        "log", "--all",
        f"--author={user_email}",
        f"--since={since_arg}",
        f"--format={log_format}"
    ])

    if not log_output:
        return []

    records = []
    for line in log_output.split("\n"):
        if not line.strip():
            continue
        parts = line.split("|||")
        if len(parts) < 3:
            continue

        commit_hash = parts[0].strip()
        message = parts[1].strip()
        iso_date = parts[2].strip()
        refs = parts[3].strip() if len(parts) > 3 else ""

        if commit_hash in existing_hashes:
            continue

        # Get changed files
        files_output = run_git(repo_path, [
            "diff-tree", "--no-commit-id", "--name-only", "-r", commit_hash
        ])
        files_changed = [f for f in files_output.split("\n") if f.strip()] if files_output else []

        # Get diff stats
        stat_output = run_git(repo_path, [
            "diff", "--stat", f"{commit_hash}~1", commit_hash
        ])
        insertions = 0
        deletions = 0
        if stat_output:
            last_line = stat_output.split("\n")[-1]
            import re
            ins_match = re.search(r'(\d+) insertion', last_line)
            del_match = re.search(r'(\d+) deletion', last_line)
            insertions = int(ins_match.group(1)) if ins_match else 0
            deletions = int(del_match.group(1)) if del_match else 0

        # Parse branch from refs
        branch = ""
        if refs:
            for ref in refs.split(","):
                ref = ref.strip()
                if ref.startswith("HEAD -> "):
                    branch = ref[8:]
                    break
        if not branch:
            branch = run_git(repo_path, ["rev-parse", "--abbrev-ref", "HEAD"])

        try:
            dt = datetime.fromisoformat(iso_date)
        except ValueError:
            dt = datetime.now(timezone.utc)

        record = {
            "ts": dt.isoformat(),
            "type": "git_commit",
            "source": "git_scan",
            "data": {
                "repo": repo_path,
                "branch": branch,
                "hash": commit_hash,
                "message": message,
                "files_changed": files_changed,
                "insertions": insertions,
                "deletions": deletions
            }
        }
        records.append(record)

    return records


def collect():
    config = load_config()
    if not config.get("collectors", {}).get("git_scan", True):
        return

    tracked_repos = config.get("tracked_repos", [])
    last_ts = config.get("git_scan_last_ts", "")

    if not tracked_repos:
        return

    now = datetime.now(timezone.utc)
    datafile = get_datafile(now)
    existing_hashes = get_existing_hashes(datafile)

    total_count = 0
    for repo_path in tracked_repos:
        repo_path = os.path.expanduser(repo_path)
        if not os.path.isdir(repo_path):
            continue

        records = scan_repo(repo_path, last_ts, existing_hashes)
        if records:
            records_by_file = {}
            for record in records:
                dt = datetime.fromisoformat(record["ts"])
                df = get_datafile(dt)
                if df not in records_by_file:
                    records_by_file[df] = []
                records_by_file[df].append(record)

            for df, recs in records_by_file.items():
                with open(df, "a") as f:
                    for rec in recs:
                        f.write(json.dumps(rec, ensure_ascii=False) + "\n")
                        existing_hashes.add(rec["data"]["hash"])

            total_count += len(records)

    config["git_scan_last_ts"] = now.isoformat()
    save_config(config)

    if total_count > 0:
        print(f"[git_scan] Found {total_count} new commits across {len(tracked_repos)} repos")


if __name__ == "__main__":
    collect()


path: collectors/collect_safari.py
content: #!/usr/bin/env python3
"""Palest Ink - Safari History Collector

Reads Safari's SQLite History database and extracts browsing records.
Note: Requires Full Disk Access permission on macOS.
"""

import json
import os
import shutil
import sqlite3
import tempfile
from datetime import datetime, timezone, timedelta

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")
SAFARI_HISTORY = os.path.expanduser("~/Library/Safari/History.db")

# Safari timestamps are seconds since 2001-01-01 (Core Data / NSDate epoch)
CORE_DATA_EPOCH = datetime(2001, 1, 1, tzinfo=timezone.utc)


def safari_time_to_datetime(safari_ts):
    """Convert Safari/Core Data timestamp to Python datetime."""
    if not safari_ts or safari_ts <= 0:
        return None
    try:
        return CORE_DATA_EPOCH + timedelta(seconds=safari_ts)
    except (OSError, ValueError, OverflowError):
        return None


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def should_exclude(url, exclude_patterns):
    for pattern in exclude_patterns:
        if url.startswith(pattern):
            return True
    return False


def collect():
    if not os.path.exists(SAFARI_HISTORY):
        return

    config = load_config()
    if not config.get("collectors", {}).get("safari", True):
        return

    last_visit_id = config.get("safari_last_visit_id", 0)
    exclude_patterns = config.get("exclude_patterns", {}).get("urls", [])

    # Copy DB to avoid lock
    tmp_fd, tmp_path = tempfile.mkstemp(suffix=".db")
    os.close(tmp_fd)
    try:
        shutil.copy2(SAFARI_HISTORY, tmp_path)
    except (PermissionError, OSError) as e:
        print(f"[safari] Cannot access Safari history: {e}")
        print("[safari] Please grant Full Disk Access to Terminal/iTerm in System Settings.")
        os.unlink(tmp_path)
        return

    try:
        conn = sqlite3.connect(tmp_path)
        conn.row_factory = sqlite3.Row

        # Safari schema: history_items (id, url, ...) + history_visits (id, history_item, visit_time, ...)
        rows = conn.execute("""
            SELECT hv.id, hi.url, hi.domain_expansion, hv.visit_time, hv.title
            FROM history_visits hv
            JOIN history_items hi ON hv.history_item = hi.id
            WHERE hv.id > ?
            ORDER BY hv.id
        """, (last_visit_id,)).fetchall()

        new_last_id = last_visit_id
        records_by_file = {}

        for row in rows:
            vid = row["id"]
            url = row["url"] or ""
            title = row["title"] or row["domain_expansion"] or ""
            vtime = row["visit_time"]

            if should_exclude(url, exclude_patterns):
                new_last_id = max(new_last_id, vid)
                continue

            dt = safari_time_to_datetime(vtime)
            if dt is None:
                new_last_id = max(new_last_id, vid)
                continue

            record = {
                "ts": dt.isoformat(),
                "type": "web_visit",
                "source": "safari_collector",
                "data": {
                    "url": url,
                    "title": title,
                    "visit_duration_seconds": 0,
                    "browser": "safari",
                    "content_pending": True
                }
            }

            datafile = get_datafile(dt)
            if datafile not in records_by_file:
                records_by_file[datafile] = []
            records_by_file[datafile].append(record)
            new_last_id = max(new_last_id, vid)

        conn.close()

        for datafile, records in records_by_file.items():
            with open(datafile, "a") as f:
                for record in records:
                    f.write(json.dumps(record, ensure_ascii=False) + "\n")

        config["safari_last_visit_id"] = new_last_id
        save_config(config)

        if rows:
            print(f"[safari] Collected {len(rows)} visits (last_id: {new_last_id})")

    finally:
        os.unlink(tmp_path)


if __name__ == "__main__":
    collect()


path: collectors/collect_shell.py
content: #!/usr/bin/env python3
"""Palest Ink - Shell History Collector

Reads zsh/bash history files and extracts command records.
"""

import json
import os
import re
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")

# Possible history file locations
ZSH_HISTORY = os.path.expanduser("~/.zsh_history")
BASH_HISTORY = os.path.expanduser("~/.bash_history")


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def should_exclude(cmd, patterns):
    for pattern in patterns:
        if re.match(pattern, cmd):
            return True
    return False


def parse_zsh_history(filepath, start_line):
    """Parse zsh extended history format: : timestamp:duration;command"""
    entries = []
    try:
        with open(filepath, "rb") as f:
            raw = f.read()
        # zsh history can have mixed encodings
        lines = raw.decode("utf-8", errors="replace").split("\n")
    except (OSError, IOError):
        return entries, start_line

    current_line = 0
    for line in lines:
        current_line += 1
        if current_line <= start_line:
            continue

        # Extended history format: : timestamp:duration;command
        match = re.match(r'^: (?P<ts>\d+):(?P<dur>\d+);(?P<cmd>.+)$', line)
        if match:
            ts = int(match.group("ts"))
            dur = int(match.group("dur"))
            cmd = match.group("cmd").strip()
            try:
                dt = datetime.fromtimestamp(ts, tz=timezone.utc)
            except (OSError, ValueError):
                continue
            entries.append((dt, cmd, dur if dur > 0 else None))
        # Multi-line commands (continuation lines start with space or no colon prefix)
        elif entries and not line.startswith(": ") and line.strip():
            # Append to previous command
            prev_dt, prev_cmd, prev_dur = entries[-1]
            entries[-1] = (prev_dt, prev_cmd + "\n" + line, prev_dur)
        # Simple format (no timestamps) — treat each non-empty line as a command
        elif not line.startswith(": ") and line.strip() and not entries:
            # First line with no timestamp — this is simple format, switch to simple parser
            return _parse_zsh_simple(lines, start_line)

    return entries, current_line


def _parse_zsh_simple(lines, start_line):
    """Parse zsh simple history format (no timestamps, one command per line)."""
    entries = []
    now = datetime.now(timezone.utc)
    current_line = 0
    for line in lines:
        current_line += 1
        if current_line <= start_line:
            continue
        cmd = line.strip()
        if cmd:
            entries.append((now, cmd, None))
    return entries, current_line


def parse_bash_history(filepath, start_line):
    """Parse bash history (simple format, one command per line).
    If HISTTIMEFORMAT is set, timestamps appear as #timestamp on preceding line.
    """
    entries = []
    try:
        with open(filepath, "r", errors="replace") as f:
            lines = f.readlines()
    except (OSError, IOError):
        return entries, start_line

    current_line = 0
    pending_ts = None
    for line in lines:
        current_line += 1
        if current_line <= start_line:
            continue

        line = line.strip()
        if line.startswith("#") and line[1:].isdigit():
            pending_ts = int(line[1:])
            continue

        if line:
            if pending_ts:
                try:
                    dt = datetime.fromtimestamp(pending_ts, tz=timezone.utc)
                except (OSError, ValueError):
                    dt = datetime.now(timezone.utc)
                pending_ts = None
            else:
                dt = datetime.now(timezone.utc)
            entries.append((dt, line, None))

    return entries, current_line


def collect():
    config = load_config()
    if not config.get("collectors", {}).get("shell", True):
        return

    last_line = config.get("shell_last_line", 0)
    exclude_patterns = config.get("exclude_patterns", {}).get("commands", [])

    entries = []
    new_last_line = last_line

    # Try zsh first (more common on macOS), then bash
    if os.path.exists(ZSH_HISTORY):
        entries, new_last_line = parse_zsh_history(ZSH_HISTORY, last_line)
    elif os.path.exists(BASH_HISTORY):
        entries, new_last_line = parse_bash_history(BASH_HISTORY, last_line)

    if not entries:
        return

    records_by_file = {}
    count = 0

    for dt, cmd, duration in entries:
        if should_exclude(cmd, exclude_patterns):
            continue
        # Skip very long commands (likely data, not real commands)
        if len(cmd) > 1000:
            continue

        record = {
            "ts": dt.isoformat(),
            "type": "shell_command",
            "source": "shell_collector",
            "data": {
                "command": cmd,
                "duration_seconds": duration,
            }
        }

        datafile = get_datafile(dt)
        if datafile not in records_by_file:
            records_by_file[datafile] = []
        records_by_file[datafile].append(record)
        count += 1

    for datafile, records in records_by_file.items():
        with open(datafile, "a") as f:
            for record in records:
                f.write(json.dumps(record, ensure_ascii=False) + "\n")

    config["shell_last_line"] = new_last_line
    save_config(config)

    if count > 0:
        print(f"[shell] Collected {count} commands")


if __name__ == "__main__":
    collect()


path: collectors/collect_vscode.py
content: #!/usr/bin/env python3
"""Palest Ink - VS Code Edit History Collector

Reads VS Code's state database to track recently opened/edited files.
"""

import json
import os
import sqlite3
import tempfile
import shutil
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")

# VS Code state database location on macOS
VSCODE_STATE_DB = os.path.expanduser(
    "~/Library/Application Support/Code/User/globalStorage/state.vscdb"
)


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def save_config(config):
    with open(CONFIG_FILE, "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)


def get_datafile(dt):
    path = os.path.join(DATA_DIR, dt.strftime("%Y"), dt.strftime("%m"), f"{dt.strftime('%d')}.jsonl")
    os.makedirs(os.path.dirname(path), exist_ok=True)
    return path


def guess_language(filepath):
    """Guess programming language from file extension."""
    ext_map = {
        ".py": "python", ".js": "javascript", ".ts": "typescript",
        ".tsx": "typescript-react", ".jsx": "javascript-react",
        ".java": "java", ".go": "go", ".rs": "rust", ".rb": "ruby",
        ".php": "php", ".c": "c", ".cpp": "cpp", ".h": "c-header",
        ".cs": "csharp", ".swift": "swift", ".kt": "kotlin",
        ".scala": "scala", ".r": "r", ".R": "r",
        ".sql": "sql", ".sh": "shell", ".bash": "shell", ".zsh": "shell",
        ".html": "html", ".css": "css", ".scss": "scss", ".less": "less",
        ".json": "json", ".yaml": "yaml", ".yml": "yaml", ".toml": "toml",
        ".xml": "xml", ".md": "markdown", ".rst": "restructuredtext",
        ".vue": "vue", ".svelte": "svelte",
    }
    _, ext = os.path.splitext(filepath)
    return ext_map.get(ext.lower(), "unknown")


def collect():
    if not os.path.exists(VSCODE_STATE_DB):
        return

    config = load_config()
    if not config.get("collectors", {}).get("vscode", True):
        return

    last_ts = config.get("vscode_last_ts", 0)

    # Copy DB to avoid lock
    tmp_fd, tmp_path = tempfile.mkstemp(suffix=".db")
    os.close(tmp_fd)
    try:
        shutil.copy2(VSCODE_STATE_DB, tmp_path)
    except (PermissionError, OSError):
        os.unlink(tmp_path)
        return

    try:
        conn = sqlite3.connect(tmp_path)
        conn.row_factory = sqlite3.Row

        # VS Code stores state as key-value pairs
        # Key: "history.recentlyOpenedPathsList" contains recently opened files/workspaces
        row = conn.execute(
            "SELECT value FROM ItemTable WHERE key = ?",
            ("history.recentlyOpenedPathsList",)
        ).fetchone()

        if not row:
            conn.close()
            return

        recent_data = json.loads(row["value"])
        entries = recent_data.get("entries", [])

        now = datetime.now(timezone.utc)
        datafile = get_datafile(now)
        count = 0

        # Also try to get file modification times for better timestamps
        new_entries = []
        for entry in entries:
            file_uri = entry.get("fileUri") or entry.get("folderUri") or ""
            # Convert file URI to path
            if file_uri.startswith("file://"):
                filepath = file_uri[7:]  # Remove file:// prefix
            else:
                continue

            # Get file modification time as proxy for "edit time"
            try:
                mtime = os.path.getmtime(filepath)
            except OSError:
                continue

            # Only record files modified since last check
            if mtime <= last_ts:
                continue

            dt = datetime.fromtimestamp(mtime, tz=timezone.utc)
            workspace = ""
            # Try to find the workspace/project root
            for parent in [os.path.dirname(filepath)]:
                if os.path.exists(os.path.join(parent, ".git")):
                    workspace = parent
                    break

            record = {
                "ts": dt.isoformat(),
                "type": "vscode_edit",
                "source": "vscode_collector",
                "data": {
                    "file_path": filepath,
                    "workspace": workspace,
                    "language": guess_language(filepath),
                    "is_folder": entry.get("folderUri") is not None
                }
            }
            new_entries.append((mtime, record))
            count += 1

        conn.close()

        if new_entries:
            # Group by date
            records_by_file = {}
            max_ts = last_ts
            for mtime, record in new_entries:
                dt = datetime.fromisoformat(record["ts"])
                df = get_datafile(dt)
                if df not in records_by_file:
                    records_by_file[df] = []
                records_by_file[df].append(record)
                max_ts = max(max_ts, mtime)

            for df, records in records_by_file.items():
                with open(df, "a") as f:
                    for record in records:
                        f.write(json.dumps(record, ensure_ascii=False) + "\n")

            config["vscode_last_ts"] = max_ts
            save_config(config)
            print(f"[vscode] Collected {count} file edits")

    finally:
        os.unlink(tmp_path)


if __name__ == "__main__":
    collect()


path: collectors/cron_collect.sh
content: #!/bin/bash
# Palest Ink - Cron Collection Entry Point
# Runs all collectors in sequence: browsers first, then content, then others

PALEST_INK_DIR="$HOME/.palest-ink"
BIN_DIR="$PALEST_INK_DIR/bin"
LOCKFILE="$PALEST_INK_DIR/tmp/collect.lock"

# Prevent concurrent runs
if [ -f "$LOCKFILE" ]; then
    LOCK_PID=$(cat "$LOCKFILE" 2>/dev/null)
    if kill -0 "$LOCK_PID" 2>/dev/null; then
        echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Collection already running (PID $LOCK_PID), skipping."
        exit 0
    fi
    # Stale lock, remove it
    rm -f "$LOCKFILE"
fi

echo $$ > "$LOCKFILE"
trap "rm -f '$LOCKFILE'" EXIT

echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Starting collection..."

# Phase 0: App focus (fastest, highest priority)
timeout 5 python3 "$BIN_DIR/collect_app.py" 2>&1 || true

# Phase 1: Browser history (creates web_visit records with content_pending=true)
python3 "$BIN_DIR/collect_chrome.py" 2>&1 || true
python3 "$BIN_DIR/collect_safari.py" 2>&1 || true

# Phase 2: Content fetching (fills in content_summary for pending URLs)
python3 "$BIN_DIR/collect_content.py" 2>&1 || true

# Phase 3: Other collectors
python3 "$BIN_DIR/collect_shell.py" 2>&1 || true
python3 "$BIN_DIR/collect_vscode.py" 2>&1 || true
python3 "$BIN_DIR/collect_git.py" 2>&1 || true

# Phase 4: Filesystem events (find can be slow, run last)
timeout 15 python3 "$BIN_DIR/collect_fsevent.py" 2>&1 || true

echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] Collection complete."

# Size check: warn when data directory approaches 2 GB threshold
DATA_DIR="$PALEST_INK_DIR/data"
FLAG_FILE="$PALEST_INK_DIR/tmp/cleanup_needed"
if [ -d "$DATA_DIR" ]; then
    DATA_SIZE_KB=$(du -sk "$DATA_DIR" 2>/dev/null | cut -f1)
    DATA_SIZE_BYTES=$((DATA_SIZE_KB * 1024))
    WARN_THRESHOLD=$((1800 * 1024 * 1024))   # 1.8 GB
    if [ "$DATA_SIZE_BYTES" -gt "$WARN_THRESHOLD" ]; then
        echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] WARNING: data directory is ${DATA_SIZE_KB}KB, approaching 2 GB limit."
        printf '{"size_bytes":%d,"size_human":"%s MB","ts":"%s"}' \
            "$DATA_SIZE_BYTES" \
            "$((DATA_SIZE_KB / 1024))" \
            "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
            > "$FLAG_FILE"
    fi
fi


path: collectors/git_hook.sh
content: #!/bin/bash
# Palest Ink - Git Hook Script
# This script is installed as post-commit, post-merge, post-checkout, pre-push

PALEST_INK_DIR="$HOME/.palest-ink"
HOOK_NAME="$(basename "$0")"
DATE_DIR="$(date +%Y)/$(date +%m)"
DATAFILE="$PALEST_INK_DIR/data/$DATE_DIR/$(date +%d).jsonl"
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Ensure data directory exists
mkdir -p "$(dirname "$DATAFILE")"

# Helper: escape string for JSON
json_escape() {
    python3 -c "import json,sys; print(json.dumps(sys.stdin.read().strip()))" 2>/dev/null <<< "$1"
}

case "$HOOK_NAME" in
    post-commit)
        REPO=$(git rev-parse --show-toplevel 2>/dev/null || echo "unknown")
        HASH=$(git rev-parse HEAD 2>/dev/null || echo "unknown")
        BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")
        MESSAGE=$(git log -1 --pretty=%s 2>/dev/null || echo "")
        FILES_JSON=$(git diff-tree --no-commit-id --name-only -r HEAD 2>/dev/null | python3 -c "import sys,json; print(json.dumps([l.strip() for l in sys.stdin if l.strip()]))" 2>/dev/null || echo "[]")
        STATS=$(git diff --stat HEAD~1 HEAD 2>/dev/null | tail -1 || echo "")
        INSERTIONS=$(echo "$STATS" | grep -oE '[0-9]+ insertion' | grep -oE '[0-9]+' || echo "0")
        DELETIONS=$(echo "$STATS" | grep -oE '[0-9]+ deletion' | grep -oE '[0-9]+' || echo "0")

        python3 -c "
import json
record = {
    'ts': '$TIMESTAMP',
    'type': 'git_commit',
    'source': 'git_hook',
    'data': {
        'repo': '$REPO',
        'branch': '$BRANCH',
        'hash': '$HASH',
        'message': $(json_escape "$MESSAGE"),
        'files_changed': $FILES_JSON,
        'insertions': int('${INSERTIONS:-0}'),
        'deletions': int('${DELETIONS:-0}')
    }
}
with open('$DATAFILE', 'a') as f:
    f.write(json.dumps(record, ensure_ascii=False) + '\n')
" 2>/dev/null
        ;;

    post-merge)
        REPO=$(git rev-parse --show-toplevel 2>/dev/null || echo "unknown")
        BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")
        # $1 is squash flag
        IS_SQUASH="${1:-0}"

        python3 -c "
import json
record = {
    'ts': '$TIMESTAMP',
    'type': 'git_pull',
    'source': 'git_hook',
    'data': {
        'repo': '$REPO',
        'branch': '$BRANCH',
        'is_squash': bool(int('$IS_SQUASH'))
    }
}
with open('$DATAFILE', 'a') as f:
    f.write(json.dumps(record, ensure_ascii=False) + '\n')
" 2>/dev/null
        ;;

    post-checkout)
        REPO=$(git rev-parse --show-toplevel 2>/dev/null || echo "unknown")
        PREV_HEAD="$1"
        NEW_HEAD="$2"
        BRANCH_FLAG="$3"  # 1 = branch checkout, 0 = file checkout

        # Only record branch checkouts
        if [ "$BRANCH_FLAG" = "1" ]; then
            FROM_BRANCH=$(git name-rev --name-only "$PREV_HEAD" 2>/dev/null || echo "unknown")
            TO_BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")

            python3 -c "
import json
record = {
    'ts': '$TIMESTAMP',
    'type': 'git_checkout',
    'source': 'git_hook',
    'data': {
        'repo': '$REPO',
        'from_ref': '$FROM_BRANCH',
        'to_branch': '$TO_BRANCH'
    }
}
with open('$DATAFILE', 'a') as f:
    f.write(json.dumps(record, ensure_ascii=False) + '\n')
" 2>/dev/null
        fi
        ;;

    pre-push)
        REPO=$(git rev-parse --show-toplevel 2>/dev/null || echo "unknown")
        BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")
        REMOTE_NAME="$1"
        REMOTE_URL="$2"

        python3 -c "
import json
record = {
    'ts': '$TIMESTAMP',
    'type': 'git_push',
    'source': 'git_hook',
    'data': {
        'repo': '$REPO',
        'branch': '$BRANCH',
        'remote': '$REMOTE_NAME',
        'remote_url': '$REMOTE_URL'
    }
}
with open('$DATAFILE', 'a') as f:
    f.write(json.dumps(record, ensure_ascii=False) + '\n')
" 2>/dev/null
        ;;
esac

# Chain to previous global hooks directory (backed up during install)
PREV_HOOKS_PATH=""
if [ -f "$PALEST_INK_DIR/config.json" ]; then
    PREV_HOOKS_PATH=$(python3 -c "
import json
with open('$PALEST_INK_DIR/config.json') as f:
    print(json.load(f).get('previous_hooks_path', ''))
" 2>/dev/null || echo "")
fi
if [ -n "$PREV_HOOKS_PATH" ] && [ -x "$PREV_HOOKS_PATH/$HOOK_NAME" ]; then
    "$PREV_HOOKS_PATH/$HOOK_NAME" "$@"
fi

# Also chain to project-local hook if it exists
LOCAL_HOOK="$(git rev-parse --git-dir 2>/dev/null)/hooks/${HOOK_NAME}.local"
if [ -x "$LOCAL_HOOK" ]; then
    exec "$LOCAL_HOOK" "$@"
fi

exit 0


path: collectors/install.sh
content: #!/bin/bash
# Palest Ink (淡墨) - Installation Script
# 好记性不如烂笔头

set -e

PALEST_INK_DIR="$HOME/.palest-ink"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"

GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RED='\033[0;31m'
NC='\033[0m'

info() { echo -e "${GREEN}[palest-ink]${NC} $1"; }
warn() { echo -e "${YELLOW}[palest-ink]${NC} $1"; }
error() { echo -e "${RED}[palest-ink]${NC} $1"; }

# Step 1: Create data directories
info "Creating data directories..."
mkdir -p "$PALEST_INK_DIR"/{data,reports,hooks,bin,tmp}

# Step 2: Write default config if not exists
if [ ! -f "$PALEST_INK_DIR/config.json" ]; then
    info "Writing default configuration..."
    cat > "$PALEST_INK_DIR/config.json" << 'CONFIGEOF'
{
  "version": 1,
  "chrome_last_visit_id": 0,
  "safari_last_visit_id": 0,
  "shell_last_line": 0,
  "vscode_last_ts": 0,
  "git_scan_last_ts": "",
  "tracked_repos": [],
  "app_last_app": "",
  "app_last_window": "",
  "app_last_ts": "",
  "app_last_record_line": -1,
  "collectors": {
    "chrome": true,
    "safari": true,
    "shell": true,
    "vscode": true,
    "git_hooks": true,
    "git_scan": true,
    "content": true,
    "app": true,
    "fsevent": true
  },
  "exclude_patterns": {
    "urls": [
      "chrome://",
      "chrome-extension://",
      "about:",
      "file://"
    ],
    "commands": [
      "^ls$",
      "^cd ",
      "^clear$",
      "^pwd$",
      "^exit$",
      "^history"
    ]
  },
  "content_fetch": {
    "max_urls_per_run": 50,
    "summary_max_chars": 800,
    "timeout_seconds": 10
  },
  "app": {
    "min_focus_seconds": 600,
    "exclude": ["loginwindow", "Dock", "SystemUIServer", "Finder", "ScreenSaverEngine"]
  },
  "app_categories": {
    "development": ["Cursor", "Code", "Xcode", "Terminal", "iTerm2", "Warp"],
    "browser": ["Google Chrome", "Safari", "Firefox", "Arc"],
    "communication": ["Slack", "WeChat", "Discord", "Zoom"]
  },
  "watched_dirs": []
}
CONFIGEOF
else
    info "Configuration already exists, skipping."
fi

# Step 3: Install git hooks
info "Installing git hooks..."

# Backup existing core.hooksPath if set
EXISTING_HOOKS_PATH=$(git config --global core.hooksPath 2>/dev/null || echo "")
if [ -n "$EXISTING_HOOKS_PATH" ] && [ "$EXISTING_HOOKS_PATH" != "$PALEST_INK_DIR/hooks" ]; then
    warn "Existing global hooksPath found: $EXISTING_HOOKS_PATH"
    warn "Backing up to config.json"
    python3 -c "
import json
with open('$PALEST_INK_DIR/config.json', 'r') as f:
    cfg = json.load(f)
cfg['previous_hooks_path'] = '$EXISTING_HOOKS_PATH'
with open('$PALEST_INK_DIR/config.json', 'w') as f:
    json.dump(cfg, f, indent=2)
"
fi

# Copy hook scripts
for hook in post-commit post-merge post-checkout pre-push; do
    cp "$SCRIPT_DIR/git_hook.sh" "$PALEST_INK_DIR/hooks/$hook"
    chmod +x "$PALEST_INK_DIR/hooks/$hook"
done

git config --global core.hooksPath "$PALEST_INK_DIR/hooks"
info "Git hooks installed (core.hooksPath -> $PALEST_INK_DIR/hooks)"

# Step 4: Copy collector scripts to bin
info "Copying collector scripts..."
cp "$SCRIPT_DIR"/collect_*.py "$PALEST_INK_DIR/bin/"
cp "$SCRIPT_DIR/cleanup.py" "$PALEST_INK_DIR/bin/"
cp "$SCRIPT_DIR/cron_collect.sh" "$PALEST_INK_DIR/bin/"
chmod +x "$PALEST_INK_DIR/bin/cron_collect.sh"

# Step 5: Install launchd agent (every 15 seconds)
info "Setting up launchd agent (every 15 seconds)..."
PLIST_LABEL="com.palest-ink.collector"
PLIST_PATH="$HOME/Library/LaunchAgents/${PLIST_LABEL}.plist"

# Remove old cron job if present
if crontab -l 2>/dev/null | grep -q "palest-ink"; then
    warn "Removing old cron job..."
    crontab -l 2>/dev/null | grep -v "palest-ink" | crontab -
fi

# Unload existing plist if present
if launchctl list | grep -q "$PLIST_LABEL" 2>/dev/null; then
    launchctl unload "$PLIST_PATH" 2>/dev/null || true
fi

cat > "$PLIST_PATH" << PLISTEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>${PLIST_LABEL}</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>${PALEST_INK_DIR}/bin/cron_collect.sh</string>
    </array>
    <key>StartInterval</key>
    <integer>15</integer>
    <key>StandardOutPath</key>
    <string>${PALEST_INK_DIR}/cron.log</string>
    <key>StandardErrorPath</key>
    <string>${PALEST_INK_DIR}/cron.log</string>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>
PLISTEOF

launchctl load "$PLIST_PATH"
info "launchd agent installed (every 15 seconds)."

# Step 6: Check permissions for Safari
if [ -f "$HOME/Library/Safari/History.db" ]; then
    if ! cp "$HOME/Library/Safari/History.db" "$PALEST_INK_DIR/tmp/safari_test.db" 2>/dev/null; then
        warn "Cannot access Safari history. Please grant Full Disk Access to Terminal:"
        warn "  System Settings > Privacy & Security > Full Disk Access > Terminal.app"
    else
        rm -f "$PALEST_INK_DIR/tmp/safari_test.db"
        info "Safari history access: OK"
    fi
fi

# Step 6b: Accessibility permission notice for app focus tracking
warn ""
warn "App Focus Tracking requires Accessibility permission:"
warn "  System Settings > Privacy & Security > Accessibility > Terminal.app (enable)"
warn "Without this, collect_app.py will not be able to detect the frontmost app."

# Step 7: Check Chrome
if [ -f "$HOME/Library/Application Support/Google/Chrome/Default/History" ]; then
    info "Chrome history found: OK"
else
    warn "Chrome history not found (Chrome may not be installed)"
fi

# Step 8: Record project path for skill reference
echo "$PROJECT_DIR" > "$PALEST_INK_DIR/project_path"

info ""
info "============================================"
info "  Palest Ink installed successfully! 淡墨"
info "============================================"
info ""
info "Data directory: $PALEST_INK_DIR"
info "Git hooks: active (post-commit, post-merge, post-checkout, pre-push)"
info "launchd agent: every 15 seconds"
info ""
info "Try making a git commit — then check:"
info "  cat $PALEST_INK_DIR/data/\$(date +%Y)/\$(date +%m)/\$(date +%d).jsonl"


path: collectors/uninstall.sh
content: #!/bin/bash
# Palest Ink (淡墨) - Uninstallation Script

set -e

PALEST_INK_DIR="$HOME/.palest-ink"

GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

info() { echo -e "${GREEN}[palest-ink]${NC} $1"; }
warn() { echo -e "${YELLOW}[palest-ink]${NC} $1"; }

# Step 1: Remove launchd agent
PLIST_LABEL="com.palest-ink.collector"
PLIST_PATH="$HOME/Library/LaunchAgents/${PLIST_LABEL}.plist"
info "Removing launchd agent..."
if launchctl list | grep -q "$PLIST_LABEL" 2>/dev/null; then
    launchctl unload "$PLIST_PATH" 2>/dev/null || true
    info "launchd agent unloaded."
fi
if [ -f "$PLIST_PATH" ]; then
    rm -f "$PLIST_PATH"
    info "launchd plist removed."
else
    info "No launchd plist found."
fi

# Also remove legacy cron job if present
if crontab -l 2>/dev/null | grep -q "palest-ink"; then
    crontab -l 2>/dev/null | grep -v "palest-ink" | crontab -
    info "Legacy cron job removed."
fi

# Step 2: Restore git hooks path
info "Restoring git hooks..."
CURRENT_HOOKS=$(git config --global core.hooksPath 2>/dev/null || echo "")
if [ "$CURRENT_HOOKS" = "$PALEST_INK_DIR/hooks" ]; then
    # Check if there was a previous hooks path
    if [ -f "$PALEST_INK_DIR/config.json" ]; then
        PREV_PATH=$(python3 -c "
import json
with open('$PALEST_INK_DIR/config.json') as f:
    cfg = json.load(f)
print(cfg.get('previous_hooks_path', ''))
" 2>/dev/null || echo "")
        if [ -n "$PREV_PATH" ]; then
            git config --global core.hooksPath "$PREV_PATH"
            info "Restored previous hooksPath: $PREV_PATH"
        else
            git config --global --unset core.hooksPath
            info "Removed global hooksPath."
        fi
    else
        git config --global --unset core.hooksPath
        info "Removed global hooksPath."
    fi
fi

# Step 3: Ask about data
echo ""
warn "Data directory: $PALEST_INK_DIR"
read -p "Delete all collected data? (y/N): " DELETE_DATA
if [ "$DELETE_DATA" = "y" ] || [ "$DELETE_DATA" = "Y" ]; then
    rm -rf "$PALEST_INK_DIR"
    info "Data directory removed."
else
    # Only remove hooks and bin, keep data
    rm -rf "$PALEST_INK_DIR/hooks" "$PALEST_INK_DIR/bin"
    info "Hooks and scripts removed. Data preserved at $PALEST_INK_DIR/data/"
fi

info ""
info "Palest Ink uninstalled."


path: references/schema.md
content: # Palest Ink Data Schema

## Storage Location

All activity data is stored at `~/.palest-ink/data/YYYY/MM/DD.jsonl`.
Each file contains one day's activities, one JSON record per line.

## Record Format

Every record has the same top-level structure:

```json
{
  "ts": "2026-03-03T14:22:31Z",
  "type": "git_commit",
  "source": "git_hook",
  "data": { ... }
}
```

| Field | Type | Description |
|-------|------|-------------|
| `ts` | string | ISO 8601 timestamp |
| `type` | string | Activity type (see below) |
| `source` | string | Which collector produced this record |
| `data` | object | Type-specific data fields |

## Activity Types

### git_commit
Source: `git_hook` or `git_scan`

```json
{
  "repo": "/Users/xuyun/my-project",
  "branch": "main",
  "hash": "a1b2c3d",
  "message": "Fix login validation bug",
  "files_changed": ["src/auth.py", "tests/test_auth.py"],
  "insertions": 42,
  "deletions": 15
}
```

### git_push
Source: `git_hook`

```json
{
  "repo": "/Users/xuyun/my-project",
  "branch": "main",
  "remote": "origin",
  "remote_url": "[REDACTED_SECRET]:user/repo.git"
}
```

### git_pull
Source: `git_hook`

```json
{
  "repo": "/Users/xuyun/my-project",
  "branch": "main",
  "is_squash": false
}
```

### git_checkout
Source: `git_hook`

```json
{
  "repo": "/Users/xuyun/my-project",
  "from_ref": "main",
  "to_branch": "feature/auth"
}
```

### web_visit
Source: `chrome_collector` or `safari_collector`

```json
{
  "url": "https://brew.sh/",
  "title": "Homebrew — The Missing Package Manager for macOS",
  "visit_duration_seconds": 120,
  "browser": "chrome",
  "content_summary": "Homebrew installs the stuff you need...",
  "content_keywords": ["homebrew", "package manager", "macOS", "install"],
  "content_pending": false
}
```

**Content fields:**
- `content_pending`: `true` if content hasn't been fetched yet
- `content_summary`: First 800 chars of extracted page text
- `content_keywords`: Top 10 keywords extracted from page content
- `content_error`: `true` if content fetch failed

### shell_command
Source: `shell_collector`

```json
{
  "command": "git log --oneline -5"
}
```

### vscode_edit
Source: `vscode_collector`

```json
{
  "file_path": "/Users/xuyun/project/src/main.py",
  "workspace": "/Users/xuyun/project",
  "language": "python",
  "is_folder": false
}
```

### app_focus
Source: `app_collector`

```json
{
  "app_name": "Cursor",
  "window_title": "palest_ink — main.py",
  "duration_seconds": 45
}
```

Records the frontmost application and window. When the same app+window is detected in consecutive collection cycles, `duration_seconds` is accumulated in-place rather than creating new records.

### file_change
Source: `fsevent_collector`

```json
{
  "path": "/Users/xuyun/project/src/main.py",
  "workspace": "/Users/xuyun/project",
  "language": "python",
  "event": "modified"
}
```

Detected via `find -newer <marker>` on directories listed in `watched_dirs` (or `tracked_repos` as fallback). The `workspace` field is the nearest parent directory containing a `.git` folder.

### shell_command (enhanced)
Source: `shell_collector`

```json
{
  "command": "git log --oneline -5",
  "duration_seconds": 2
}
```

`duration_seconds` is extracted from the zsh extended history format (`: timestamp:duration;command`). Set to `null` for bash or simple-format zsh history.

## Configuration

Config file: `~/.palest-ink/config.json`

Key fields:
- `collectors`: Enable/disable individual collectors (including `app` and `fsevent`)
- `tracked_repos`: List of git repo paths for git_scan collector
- `watched_dirs`: Directories to monitor for file changes (falls back to `tracked_repos` if empty)
- `exclude_patterns.urls`: URL prefixes to ignore
- `exclude_patterns.commands`: Regex patterns for commands to ignore
- `content_fetch.max_urls_per_run`: Max URLs to fetch per collection cycle
- `content_fetch.summary_max_chars`: Max chars for content summary
- `app.min_focus_seconds`: Minimum session duration to show in Focus Sessions report (default: 600)
- `app.exclude`: App names to skip during app focus collection
- `app_categories`: Map of category names to app name lists for usage grouping
- `app_last_app`, `app_last_window`, `app_last_ts`, `app_last_record_line`: State for app focus in-place duration merging


path: scripts/status.py
content: #!/usr/bin/env python3
"""Palest Ink - Status Checker

Shows collection status, data volume, and system health.

Usage:
    python3 status.py
"""

import json
import os
import subprocess
from datetime import datetime, timezone

PALEST_INK_DIR = os.path.expanduser("~/.palest-ink")
CONFIG_FILE = os.path.join(PALEST_INK_DIR, "config.json")
DATA_DIR = os.path.join(PALEST_INK_DIR, "data")
CLEANUP_FLAG = os.path.join(PALEST_INK_DIR, "tmp", "cleanup_needed")


def load_config():
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, "r") as f:
            return json.load(f)
    return {}


def count_today_records():
    now = datetime.now()
    path = os.path.join(DATA_DIR, now.strftime("%Y"), now.strftime("%m"), f"{now.strftime('%d')}.jsonl")
    if not os.path.exists(path):
        return 0, {}
    counts = {}
    total = 0
    with open(path, "r") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                record = json.loads(line)
                rtype = record.get("type", "unknown")
                counts[rtype] = counts.get(rtype, 0) + 1
                total += 1
            except json.JSONDecodeError:
                pass
    return total, counts


def get_data_size():
    """Get total size of data directory."""
    total = 0
    for root, dirs, files in os.walk(DATA_DIR):
        for f in files:
            total += os.path.getsize(os.path.join(root, f))
    return total


def check_cleanup_needed():
    """Return flag info dict if cleanup is needed, else None."""
    if not os.path.exists(CLEANUP_FLAG):
        return None
    try:
        with open(CLEANUP_FLAG, "r") as f:
            return json.load(f)
    except (OSError, json.JSONDecodeError):
        return {}


def check_cron():
    """Check if cron job is installed."""
    try:
        result = subprocess.run(
            ["crontab", "-l"],
            capture_output=True, text=True, timeout=5
        )
        return "palest-ink" in result.stdout
    except (subprocess.TimeoutExpired, OSError):
        return False


def check_git_hooks():
    """Check if git hooks are configured."""
    try:
        result = subprocess.run(
            ["git", "config", "--global", "core.hooksPath"],
            capture_output=True, text=True, timeout=5
        )
        hooks_path = result.stdout.strip()
        return hooks_path == os.path.join(PALEST_INK_DIR, "hooks")
    except (subprocess.TimeoutExpired, OSError):
        return False


def format_size(size_bytes):
    for unit in ["B", "KB", "MB", "GB"]:
        if size_bytes < 1024:
            return f"{size_bytes:.1f} {unit}"
        size_bytes /= 1024
    return f"{size_bytes:.1f} TB"


def main():
    print("=" * 50)
    print("  Palest Ink (淡墨) - Status")
    print("=" * 50)
    print()

    # Data directory
    if not os.path.exists(PALEST_INK_DIR):
        print("Status: NOT INSTALLED")
        print(f"Run install.sh to set up Palest Ink.")
        return

    config = load_config()
    collectors = config.get("collectors", {})

    # Today's records
    total, counts = count_today_records()
    data_bytes = get_data_size()
    print(f"Data directory: {DATA_DIR}")
    print(f"Data size: {format_size(data_bytes)}")
    print()

    # Cleanup warning
    cleanup_flag = check_cleanup_needed()
    if cleanup_flag is not None:
        flag_size = cleanup_flag.get("size_human", "")
        size_note = f" ({flag_size} MB)" if flag_size else ""
        print("=" * 50)
        print(f"  ⚠️  CLEANUP RECOMMENDED")
        print(f"  Data{size_note} is approaching the 2 GB limit.")
        print(f"  Run cleanup to remove oldest records:")
        print(f"    python3 ~/.palest-ink/bin/cleanup.py --dry-run")
        print(f"    python3 ~/.palest-ink/bin/cleanup.py")
        print("=" * 50)
        print()

    print(f"Today's records: {total}")
    if counts:
        for rtype, count in sorted(counts.items(), key=lambda x: x[1], reverse=True):
            print(f"  {rtype}: {count}")
    print()

    # Collector status
    print("Collectors:")
    cron_active = check_cron()
    hooks_active = check_git_hooks()

    print(f"  git_hooks:  {'ACTIVE' if hooks_active else 'INACTIVE'} (core.hooksPath)")
    print(f"  cron:       {'ACTIVE' if cron_active else 'INACTIVE'} (every 3 min)")
    print(f"  chrome:     {'enabled' if collectors.get('chrome', True) else 'disabled'}")
    print(f"  safari:     {'enabled' if collectors.get('safari', True) else 'disabled'}")
    print(f"  shell:      {'enabled' if collectors.get('shell', True) else 'disabled'}")
    print(f"  vscode:     {'enabled' if collectors.get('vscode', True) else 'disabled'}")
    print(f"  git_scan:   {'enabled' if collectors.get('git_scan', True) else 'disabled'}")
    print(f"  content:    {'enabled' if collectors.get('content', True) else 'disabled'}")
    print()

    # Last cron run
    cron_log = os.path.join(PALEST_INK_DIR, "cron.log")
    if os.path.exists(cron_log):
        try:
            with open(cron_log, "r") as f:
                lines = f.readlines()
            # Find last "Starting collection" or "Collection complete"
            for line in reversed(lines):
                if "Collection complete" in line or "Starting collection" in line:
                    print(f"Last cron: {line.strip()}")
                    break
        except OSError:
            pass

    # Tracked repos
    tracked = config.get("tracked_repos", [])
    if tracked:
        print(f"\nTracked repos ({len(tracked)}):")
        for repo in tracked:
            print(f"  {repo}")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)