Skill: init-v/initv-data-pods (version 0.2.0)

--- name: data-pods description: Create and manage modular portable database pods (SQLite + metadata + embeddings). Includes document ingestion with embeddings for semantic search. Full automation - just ask. --- # Data Pods ## Overview Create and manage portable, consent-scoped database pods. Handles document ingestion with embeddings and semantic search. ## Architecture ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Ingestion │ ──► │ DB Pods │ ──► │ Generation │ │ (ingest) │ │ (storage) │ │ (query) │ └─────────────┘ └─────────────┘ └─────────────┘ ``` ## Triggers - "create a pod" / "new pod" - "list my pods" / "what pods do I have" - "add to pod" / "add note" / "add content" - "query pod" / "search pod" - "ingest documents" / "add files" - "semantic search" / "find相关内容" - "export pod" / "pack pod" ## Core Features ### 1. Create Pod When user asks to create a pod: 1. Ask for pod name and type (scholar/health/shared/projects) 2. Run: `python3 .../scripts/pod.py create <name> --type <type>` 3. Confirm creation ### 2. Add Content (Manual) When user asks to add content: 1. Ask for pod name, title, content, tags 2. Run: `python3 .../scripts/pod.py add <pod> --title "<title>" --content "<content>" --tags "<tags>"` 3. Confirm ### 3. Ingest Documents (Automated) When user wants to ingest files: 1. Ask for pod name and folder path 2. Run: `python3 .../scripts/ingest.py ingest <pod> <folder>` 3. Supports: PDF, TXT, MD, DOCX, PNG, JPG 4. Auto-embeds text (if sentence-transformers installed) ### 4. Semantic Search When user wants to search: 1. Ask for pod name and query 2. Run: `python3 .../scripts/ingest.py search <pod> "<query>"` 3. Returns ranked results with citations ### 5. Query (Basic) When user asks to search notes: 1. Run: `python3 .../scripts/pod.py query <pod> --text "<query>"` ### 6. Export When user asks to export: 1. Run: `python3 .../scripts/podsync.py pack <pod>` ## Dependencies ```bash pip install PyPDF2 python-docx pillow pytesseract sentence-transformers ``` ## Storage Location `~/.openclaw/data-pods/` ## Key Commands ```bash # Create pod python3 .../scripts/pod.py create research --type scholar # Add note python3 .../scripts/pod.py add research --title "..." --content "..." --tags "..." # Ingest folder python3 .../scripts/ingest.py ingest research ./documents/ # Semantic search python3 .../scripts/ingest.py search research "transformers" # List documents python3 .../scripts/ingest.py list research # Query notes python3 .../scripts/pod.py query research --text "..." ``` ## Notes - Ingestion auto-chunks large documents - Embeddings enable semantic search - File hash prevents duplicate ingestion - All data stored locally in SQLite

path: README.md
content: # 🧠 Data Pods

**Modular portable database pods — your AI memory, your rules.**

Data Pods are SQLite-based knowledge bases you own. Zip them up, move them anywhere, share selectively. No vendor lock-in. No cloud dependencies. Just your data, portable.

---

## Why Data Pods?

- **Portable** — Zip a pod, move to another machine, keep working
- **Owned** — SQLite file, no proprietary formats
- **Consented** — Agents only access what you permit
- **Composable** — Multiple pods (scholar, health, projects) work together
- **GDPR-friendly** — Data stays local, export anytime

---

## What's Inside a Pod?

```
my-pod/
├── data.sqlite       # Your data (notes, embeddings, custom tables)
├── metadata.json     # Pod info (name, type, created, version)
├── embeddings/      # Vector store (future)
└── manifest.yaml    # Access rules
```

---

## Quick Start

### 1. Clone

```bash
git clone https://github.com/init-v/data-pods
cd data-pods
```

### 2. Run

```bash
# List pods
python3 pod.py list

# Create a pod
python3 pod.py create my-research --type scholar

# Add a note
python3 pod.py add my-research \
  --title "Transformer Architecture" \
  --content "The transformer uses self-attention..." \
  --tags "ai,machine-learning"

# Search
python3 pod.py query my-research --text "transformer"

# Export for sharing
python3 pod.py export my-research --output my-research.zip
```

---

## v0.2 — Document Ingestion with Embeddings

```bash
# Install dependencies
pip install PyPDF2 python-docx pillow pytesseract sentence-transformers

# Ingest a folder of documents
python3 ingest.py ingest research ./documents/

# Semantic search across all documents
python3 ingest.py search research "what did I read about transformers?"

# List ingested documents
python3 ingest.py list research
```

### Supported Formats
- PDF (via PyPDF2)
- TXT, MD, Markdown
- DOCX (via python-docx)
- Images: PNG, JPG (OCR via pytesseract)

### Features
- Auto-chunks large documents for better retrieval
- Embeddings for semantic search
- File hash prevents duplicate ingestion
- Citations with source filename

---

## Commands

| Command | Description |
|---------|-------------|
| `pod.py list` | Show all pods |
| `pod.py create <name> --type <type>` | Create new pod |
| `pod.py add <pod> --title "..." --content "..."` | Add note |
| `pod.py query <pod> --text "..."` | Search notes |
| `pod.py query <pod> --sql "SELECT *..."` | Raw SQL |
| `pod.py export <pod> --output <file>` | Export as ZIP |

---

## 🔐 Consent Layer

**Agents only access what you permit.**

Data Pods include a built-in consent layer. Before any agent can query your pods, you must grant explicit permission.

### Consent Commands

```bash
# Grant an agent access to a pod
python3 consent.py grant --pod my-research --agent openclaw

# Grant with expiration (30 days)
python3 consent.py grant --pod my-research --agent openclaw --expires 30

# List all grants
python3 consent.py list

# Check if agent has access (returns 0 if yes, 1 if no)
python3 consent.py check --pod my-research --agent openclaw

# Revoke access
python3 consent.py revoke --pod my-research --agent openclaw
```

### How It Works

1. Agent requests data from a pod
2. Consent layer checks for active grant
3. If no grant → access denied
4. If expired grant → access denied
5. All queries logged for audit

### Storage

Grants stored in `~/.config/data-pods/consents/grants.json`

---

## Pod Types

- **scholar** — Research papers, papers, embeddings
- **health** — Wearable data, biometrics (consent-only)
- **projects** — Workspace knowledge
- **shared** — Family/group data with permissions

---

## The Vision

AI agents need memory. But cloud APIs = dependence.

Data Pods give you:
1. **Sovereignty** — Your data, your machine
2. **Portability** — Move pods between devices
3. **Composability** — Multiple pods, one access layer
4. **Consent** — You control what agents see

We're building v0.1. Follow along.

---

## Roadmap

- [x] v0.1 — SQLite + notes + export
- [ ] v0.2 — Embeddings + vector search
- [ ] v0.3 — Natural language query interface
- [ ] v0.4 — OpenClaw skill integration
- [ ] v1.0 — Sync + multi-device

---

## Contributing

Open source. MIT License.

Issues welcome. PRs welcome.

---

## Contact

- X: @vinicius_init
- Discord: VR Global Enterprises (see bio)


path: REFERENCES/consent-layer-spec.md
content: # Agent Consent Layer — Technical Spec v0.1

## Vision
Personal data firewall for AI agents. You choose what pods to expose per interaction. Agents only see what you permit.

---

## Architecture

```
┌─────────────┐      ┌──────────────────┐      ┌─────────────────┐
│   User      │ ───► │  Consent Layer   │ ───► │   Data Pods     │
│ (You)       │      │  (Gateway)       │      │   (SQLite)      │
└─────────────┘      └──────────────────┘      └─────────────────┘
                            │
                            ▼
                    ┌──────────────────┐
                    │   Requesting      │
                    │   Agent          │
                    │ (External AI)    │
                    └──────────────────┘
```

---

## Components

### 1. Pod Registry
- Index all pods in `~/.openclaw/data-pods/`
- Track: name, type, created, tags, last accessed
- JSON config: which pods exist

### 2. Consent Manager
- Per-session consent storage
- Which pods are currently allowed for this agent
- Timeout: auto-revoke after N minutes
- Audit log: what was accessed when

### 3. Access Gateway
- Intercepts agent queries
- Checks consent before allowing pod access
- Returns only allowed data

### 4. Selection UI (CLI/UI)
- List available pods
- Agent declares what it needs
- User approves/denies
- "Remember for this session" option

---

## Data Structures

### Pod Manifest
```yaml
name: scholar-2024
type: scholar
tables:
  - notes
  - embeddings
tags: [research, ai, papers]
created: 2024-01-15
version: 0.1
```

### Consent Record
```json
{
  "session_id": "abc123",
  "agent": "claude-code",
  "pods_allowed": ["scholar-2024", "projects"],
  "expires_at": "2024-01-15T10:00:00Z",
  "created_at": "2024-01-15T09:30:00Z"
}
```

### Audit Log
```json
{
  "timestamp": "2024-01-15T09:35:00Z",
  "session_id": "abc123",
  "agent": "claude-code",
  "pod": "scholar-2024",
  "query": "SELECT * FROM notes WHERE tags LIKE '%ai%'",
  "rows_returned": 12
}
```

---

## Workflows

### 1. New Agent Request
```
[Agent] → "I need context about your research"
[Consent Manager] → Check existing consent
  → If valid: pass to pods
  → If none: prompt user
```

### 2. User Approval
```
[Consent Manager] → Show available pods:
  ☑ scholar-2024 (research papers)
  ☐ health-data (wearables)
  ☐ projects (workspace)
  
User selects → [Consent Manager] stores consent → Agent receives data
```

### 3. Query Execution
```
[Agent] → "What papers on transformers?"
[Access Gateway] → 
  1. Validate session consent
  2. Filter to allowed pods only
  3. Execute query against allowed pods only
  4. Log to audit
  5. Return results
```

### 4. Consent Revocation
```
User → "Revoke access"
[Consent Manager] → Remove consent for session
[Access Gateway] → Future queries blocked
```

---

## API Design (OpenClaw Skill)

```bash
# List pods
consent list

# Show current consent state
consent status --session <id>

# Grant access
consent grant <pod1> <pod2> --session <id> --duration 30m

# Revoke
consent revoke --session <id>

# Audit
consent audit --session <id>
consent logs --today
```

---

## Security

- **No default access** — Everything opt-in
- **Time-bounded** — Consent expires
- **Audit everything** — Full traceability
- **No lateral movement** — Agent can't probe unallowed pods
- **Local only** — Data never leaves your machine

---

## Future Features

- [ ] Remember consent for trusted agents
- [ ] Pod-specific column/row filtering
- [ ] Time-limited data (e.g., "last 30 days only")
- [ ] Multi-device sync (encrypted)
- [ ] Agent reputation system

---

## Why This Matters

Current AI = all or nothing
- Give access = give everything
- Privacy impossible

Consent AI = granular control
- Agent sees only what you choose
- Privacy by design
- Trust through transparency

---

## Name Ideas

- **ConsentOS**
- **PodGate**
- **ScopeAI**
- **DataFirewall**
- **SelectiveMemory**
- **AgentWall**

---

## Next Steps

1. Build v0.1: CLI consent manager
2. Integrate with OpenClaw gateway
3. Test with one external agent
4. Add UI for pod selection
5. Publish as product


path: consent.py
content: #!/usr/bin/env python3
"""
Consent Layer CLI — Agent access control for Data Pods

Usage:
    python3 consent.py grant --pod <pod_name> --agent <agent_id> [--expires <days>]
    python3 consent.py revoke --pod <pod_name> --agent <agent_id>
    python3 consent.py list [--agent <agent_id>] [--pod <pod_name>]
    python3 consent.py check --pod <pod_name> --agent <agent_id>
"""

import json
import os
import sys
import argparse
from datetime import datetime, timedelta
from pathlib import Path

CONSENT_DIR = Path.home() / ".config" / "data-pods" / "consents"
CONSENT_DIR.mkdir(parents=True, exist_ok=True)
GRANTS_FILE = CONSENT_DIR / "grants.json"


def load_grants():
    if GRANTS_FILE.exists():
        with open(GRANTS_FILE) as f:
            return json.load(f)
    return {}


def save_grants(grants):
    with open(GRANTS_FILE, "w") as f:
        json.dump(grants, f, indent=2)


def grant(pod: str, agent: str, expires_days: int = None):
    grants = load_grants()
    key = f"{pod}:{agent}"
    
    expires = None
    if expires_days:
        expires = (datetime.now() + timedelta(days=expires_days)).isoformat()
    
    grants[key] = {
        "pod": pod,
        "agent": agent,
        "granted_at": datetime.now().isoformat(),
        "expires": expires,
        "active": True
    }
    
    save_grants(grants)
    print(f"✅ Granted {agent} access to {pod}" + (f" (expires {expires})" if expires else ""))


def revoke(pod: str, agent: str):
    grants = load_grants()
    key = f"{pod}:{agent}"
    
    if key in grants:
        grants[key]["active"] = False
        grants[key]["revoked_at"] = datetime.now().isoformat()
        save_grants(grants)
        print(f"❌ Revoked {agent} access to {pod}")
    else:
        print(f"No grant found for {pod}:{agent}")


def list_grants(agent: str = None, pod: str = None):
    grants = load_grants()
    
    for key, grant in grants.items():
        if agent and grant["agent"] != agent:
            continue
        if pod and grant["pod"] != pod:
            continue
        
        status = "✅ active" if grant.get("active") else "❌ revoked"
        expires = grant.get("expires", "never")
        print(f"{grant['pod']}:{grant['agent']} — {status} — expires: {expires}")


def check(pod: str, agent: str) -> bool:
    grants = load_grants()
    key = f"{pod}:{agent}"
    
    if key not in grants:
        return False
    
    grant = grants[key]
    if not grant.get("active"):
        return False
    
    # Check expiration
    if grant.get("expires"):
        expires = datetime.fromisoformat(grant["expires"])
        if datetime.now() > expires:
            return False
    
    return True


def main():
    parser = argparse.ArgumentParser(description="Data Pods Consent Layer")
    subparsers = parser.add_subparsers(dest="command", required=True)
    
    # grant
    grant_parser = subparsers.add_parser("grant", help="Grant agent access to pod")
    grant_parser.add_argument("--pod", required=True)
    grant_parser.add_argument("--agent", required=True)
    grant_parser.add_argument("--expires", type=int, help="Days until expiration")
    
    # revoke
    revoke_parser = subparsers.add_parser("revoke", help="Revoke agent access")
    revoke_parser.add_argument("--pod", required=True)
    revoke_parser.add_argument("--agent", required=True)
    
    # list
    list_parser = subparsers.add_parser("list", help="List grants")
    list_parser.add_argument("--agent", help="Filter by agent")
    list_parser.add_argument("--pod", help="Filter by pod")
    
    # check
    check_parser = subparsers.add_parser("check", help="Check if agent has access")
    check_parser.add_argument("--pod", required=True)
    check_parser.add_argument("--agent", required=True)
    
    args = parser.parse_args()
    
    if args.command == "grant":
        grant(args.pod, args.agent, args.expires)
    elif args.command == "revoke":
        revoke(args.pod, args.agent)
    elif args.command == "list":
        list_grants(args.agent, args.pod)
    elif args.command == "check":
        result = check(args.pod, args.agent)
        sys.exit(0 if result else 1)


if __name__ == "__main__":
    main()


path: references/usage.md
content: # Data Pods v0.1 - Usage Reference

## Quick Start

```bash
# List pods
python pod.py list

# Create a pod
python pod.py create my-research --type scholar

# Add a note
python pod.py add my-research --title "Transformer Paper" --content "..." --tags "ai,research"

# Search
python pod.py query my-research --text "transformer"

# Export
python pod.py export my-research --output my-research.zip
```

## As OpenClaw Skill

```bash
# Create pod
python /home/claudio/.openclaw/workspace/skills/data-pods/scripts/pod.py create <name> --type <type>

# Add data
python /home/claudio/.openclaw/workspace/skills/data-pods/scripts/pod.py add <pod> --title "..." --content "..."

# Query
python /home/claudio/.openclaw/workspace/skills/data-pods/scripts/pod.py query <pod> --text "..."

# Export for sharing
python /home/claudio/.openclaw/workspace/skills/data-pods/scripts/pod.py export <pod> --output <path>.zip
```

## Pod Location
Default: `~/.openclaw/data-pods/`

## Tables
- `notes`: id, title, content, tags, created_at, updated_at
- `embeddings`: id, note_id, chunk_text, embedding (future vector store)


path: scripts/consent.py
content: #!/usr/bin/env python3
"""
Consent Layer v0.1 - Agent consent management for Data Pods
Usage: consent.py <command> [options]
"""

import sqlite3
import json
import os
import sys
import argparse
from pathlib import Path
from datetime import datetime, timedelta
import uuid

CONSENT_DIR = Path.home() / ".openclaw" / "consent"
PODS_DIR = Path.home() / ".openclaw" / "data-pods"

def ensure_dir():
    CONSENT_DIR.mkdir(parents=True, exist_ok=True)
    db_path = CONSENT_DIR / "consent.db"
    if not db_path.exists():
        conn = sqlite3.connect(db_path)
        c = conn.cursor()
        c.execute('''CREATE TABLE IF NOT EXISTS sessions (
            id TEXT PRIMARY KEY,
            agent TEXT,
            pods_allowed TEXT,
            created_at TEXT,
            expires_at TEXT
        )''')
        c.execute('''CREATE TABLE IF NOT EXISTS audit (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp TEXT,
            session_id TEXT,
            pod TEXT,
            query TEXT,
            rows INTEGER
        )''')
        conn.commit()
        conn.close()

def list_pods():
    """List available pods."""
    if not PODS_DIR.exists():
        print("No pods found.")
        return []
    pods = []
    for d in PODS_DIR.iterdir():
        if d.is_dir():
            pods.append(d.name)
    return pods

def status(session_id: str = None):
    """Show consent status."""
    ensure_dir()
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    if session_id:
        c.execute("SELECT * FROM sessions WHERE id = ?", (session_id,))
        row = c.fetchone()
        if row:
            print(f"Session: {row[0]}")
            print(f"Agent: {row[1]}")
            print(f"Pods allowed: {row[2]}")
            print(f"Created: {row[3]}")
            print(f"Expires: {row[4]}")
        else:
            print(f"No session: {session_id}")
    else:
        c.execute("SELECT id, agent, pods_allowed, expires_at FROM sessions")
        rows = c.fetchall()
        if rows:
            print("Active sessions:")
            for row in rows:
                print(f"  [{row[0][:8]}...] {row[1]} -> {row[2]} (expires: {row[3]})")
        else:
            print("No active sessions.")
    conn.close()

def grant(pods: list, agent: str = "unknown", duration_minutes: int = 60):
    """Grant consent to pods for a session."""
    ensure_dir()
    session_id = str(uuid.uuid4())
    created = datetime.now()
    expires = created + timedelta(minutes=duration_minutes)
    
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("INSERT INTO sessions VALUES (?, ?, ?, ?, ?)",
              (session_id, agent, ",".join(pods), created.isoformat(), expires.isoformat()))
    conn.commit()
    conn.close()
    
    print(f"Granted {len(pods)} pods to agent '{agent}'")
    print(f"  Session: {session_id}")
    print(f"  Pods: {', '.join(pods)}")
    print(f"  Expires: {expires.strftime('%H:%M')}")
    return session_id

def revoke(session_id: str):
    """Revoke consent for a session."""
    ensure_dir()
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("DELETE FROM sessions WHERE id = ?", (session_id,))
    conn.commit()
    conn.close()
    print(f"Revoked session: {session_id}")

def audit_logs(session_id: str = None, today: bool = False):
    """Show audit logs."""
    ensure_dir()
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    if session_id:
        c.execute("SELECT * FROM audit WHERE session_id = ?", (session_id,))
    else:
        c.execute("SELECT * FROM audit ORDER BY timestamp DESC LIMIT 20")
    
    rows = c.fetchall()
    if rows:
        for row in rows:
            print(f"[{row[1]}] {row[3]}: {row[4][:50]}... ({row[5]} rows)")
    else:
        print("No audit logs.")
    conn.close()

def check_consent(session_id: str, pod: str) -> bool:
    """Check if session has consent for pod."""
    ensure_dir()
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("SELECT pods_allowed, expires_at FROM sessions WHERE id = ?", (session_id,))
    row = c.fetchone()
    conn.close()
    
    if not row:
        return False
    
    pods_allowed = row[0].split(",")
    expires = datetime.fromisoformat(row[1])
    
    if datetime.now() > expires:
        return False
    
    return pod in pods_allowed

def log_access(session_id: str, pod: str, query: str, rows: int):
    """Log a pod access."""
    ensure_dir()
    db_path = CONSENT_DIR / "consent.db"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("INSERT INTO audit VALUES (NULL, ?, ?, ?, ?, ?)",
              (datetime.now().isoformat(), session_id, pod, query, rows))
    conn.commit()
    conn.close()

def main():
    parser = argparse.ArgumentParser(description="Consent Layer v0.1")
    sub = parser.add_subparsers(dest="cmd")
    
    sub.add_parser("list", help="List available pods")
    sub.add_parser("status", help="Show consent status")
    
    grant_p = sub.add_parser("grant", help="Grant consent to pods")
    grant_p.add_argument("pods", nargs="+", help="Pod names to allow")
    grant_p.add_argument("--agent", default="cli", help="Agent name")
    grant_p.add_argument("--duration", type=int, default=60, help="Minutes until expiry")
    
    revoke_p = sub.add_parser("revoke", help="Revoke consent")
    revoke_p.add_argument("session", help="Session ID")
    
    audit_p = sub.add_parser("audit", help="Show audit logs")
    audit_p.add_argument("--session", help="Filter by session")
    
    args = parser.parse_args()
    
    if args.cmd == "list":
        pods = list_pods()
        if pods:
            print("Available pods:")
            for p in pods:
                print(f"  - {p}")
        else:
            print("No pods found.")
    elif args.cmd == "status":
        status()
    elif args.cmd == "grant":
        grant(args.pods, args.agent, args.duration)
    elif args.cmd == "revoke":
        revoke(args.session)
    elif args.cmd == "audit":
        audit_logs(args.session)
    else:
        parser.print_help()

if __name__ == "__main__":
    main()


path: scripts/ingest.py
content: #!/usr/bin/env python3
"""
Data Pods Ingestion v0.2 - Document ingestion with embeddings
Usage: python ingest.py <command> [options]
"""

import sqlite3
import json
import os
import sys
import argparse
from pathlib import Path
from datetime import datetime
import hashlib
import subprocess

PODS_DIR = Path.home() / ".openclaw" / "data-pods"

# Try importing optional dependencies
try:
    import PyPDF2
    PDF_AVAILABLE = True
except ImportError:
    PDF_AVAILABLE = False

try:
    from docx import Document
    DOCX_AVAILABLE = True
except ImportError:
    DOCX_AVAILABLE = False

try:
    from PIL import Image
    import pytesseract
    OCR_AVAILABLE = True
except ImportError:
    OCR_AVAILABLE = False

try:
    from sentence_transformers import SentenceTransformer
    EMBEDDINGS_AVAILABLE = True
except ImportError:
    EMBEDDINGS_AVAILABLE = False

def ensure_pod(pod_name: str) -> Path:
    """Ensure pod exists."""
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return None
    return pod_path

def init_documents_table(pod_path: Path):
    """Ensure documents table exists."""
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    c.execute('''CREATE TABLE IF NOT EXISTS documents (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        filename TEXT,
        file_type TEXT,
        content TEXT,
        file_hash TEXT,
        chunks TEXT,
        embedding BLOB,
        created_at TEXT,
        updated_at TEXT
    )''')
    
    conn.commit()
    conn.close()

def extract_text_from_file(file_path: Path) -> str:
    """Extract text from various file types."""
    content = ""
    ext = file_path.suffix.lower()
    
    try:
        if ext == '.pdf' and PDF_AVAILABLE:
            with open(file_path, 'rb') as f:
                reader = PyPDF2.PdfReader(f)
                for page in reader.pages:
                    content += page.extract_text() + "\n"
                    
        elif ext in ['.txt', '.md', '.markdown', '.json']:
            content = file_path.read_text(encoding='utf-8', errors='ignore')
            
        elif ext == '.docx' and DOCX_AVAILABLE:
            doc = Document(file_path)
            for para in doc.paragraphs:
                content += para.text + "\n"
                
        elif ext in ['.png', '.jpg', '.jpeg', '.gif', '.bmp'] and OCR_AVAILABLE:
            image = Image.open(file_path)
            content = pytesseract.image_to_string(image)
            
        else:
            # Try as plain text
            try:
                content = file_path.read_text(encoding='utf-8', errors='ignore')
            except:
                print(f"  Warning: Cannot extract text from {file_path}")
                return ""
                
    except Exception as e:
        print(f"  Error extracting {file_path}: {e}")
        return ""
    
    return content

def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 100) -> list:
    """Split text into overlapping chunks."""
    if len(text) <= chunk_size:
        return [text]
    
    chunks = []
    start = 0
    
    while start < len(text):
        end = start + chunk_size
        
        # Try to break at word boundary
        if end < len(text):
            last_space = text.rfind(' ', start, end)
            if last_space > start:
                end = last_space
        
        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        
        start = end - overlap
        if start >= len(text):
            break
    
    return chunks

def get_file_hash(file_path: Path) -> str:
    """Get SHA256 hash of file."""
    h = hashlib.sha256()
    with open(file_path, 'rb') as f:
        h.update(f.read())
    return h.hexdigest()[:16]

def generate_embedding(text: str, model_name: str = 'all-MiniLM-L6-v2'):
    """Generate embedding for text."""
    if not EMBEDDINGS_AVAILABLE:
        return None
    
    try:
        model = SentenceTransformer(model_name)
        embedding = model.encode(text, convert_to_numpy=True)
        return embedding.tobytes()
    except Exception as e:
        print(f"  Error generating embedding: {e}")
        return None

def cosine_similarity(a: bytes, b: bytes) -> float:
    """Calculate cosine similarity between two embeddings."""
    import numpy as np
    a = np.frombuffer(a, dtype=np.float32)
    b = np.frombuffer(b, dtype=np.float32)
    
    dot = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    
    if norm_a == 0 or norm_b == 0:
        return 0.0
    
    return float(dot / (norm_a * norm_b))

def ingest_folder(pod_name: str, folder_path: str, recursive: bool = True):
    """Ingest all files from a folder."""
    pod_path = ensure_pod(pod_name)
    if not pod_path:
        return False
    
    init_documents_table(pod_path)
    folder = Path(folder_path)
    
    if not folder.exists():
        print(f"Error: Folder '{folder}' not found")
        return False
    
    # Find all supported files
    patterns = ['*.pdf', '*.txt', '*.md', '*.markdown', '*.docx', '*.png', '*.jpg', '*.jpeg']
    files = []
    for pattern in patterns:
        if recursive:
            files.extend(folder.rglob(pattern))
        else:
            files.extend(folder.glob(pattern))
    
    print(f"Found {len(files)} files to ingest")
    
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    ingested = 0
    skipped = 0
    
    for file_path in files:
        print(f"Processing: {file_path.name}")
        
        # Check if already ingested (by hash)
        file_hash = get_file_hash(file_path)
        c.execute("SELECT id FROM documents WHERE file_hash = ?", (file_hash,))
        if c.fetchone():
            print(f"  Skipped (already exists): {file_hash}")
            skipped += 1
            continue
        
        # Extract text
        content = extract_text_from_file(file_path)
        if not content:
            print(f"  Skipped (no content)")
            skipped += 1
            continue
        
        # Chunk text
        chunks = chunk_text(content)
        
        # Generate embedding
        embedding = None
        if EMBEDDINGS_AVAILABLE:
            # Embed first chunk (or combine)
            combined = " | ".join(chunks[:3])  # First 3 chunks
            embedding = generate_embedding(combined)
            print(f"  Embedded: {len(chunks)} chunks")
        else:
            print(f"  No embeddings (sentence-transformers not installed)")
        
        # Store in DB
        now = datetime.now().isoformat()
        c.execute("""INSERT INTO documents 
            (filename, file_type, content, file_hash, chunks, embedding, created_at, updated_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
            (file_path.name, file_path.suffix, content, file_hash, 
             json.dumps(chunks), embedding, now, now))
        
        ingested += 1
        print(f"  Added: {file_path.name}")
    
    conn.commit()
    conn.close()
    
    print(f"\nDone: {ingested} ingested, {skipped} skipped")
    return True

def search_semantic(pod_name: str, query: str, top_k: int = 5):
    """Semantic search across documents."""
    pod_path = ensure_pod(pod_name)
    if not pod_path:
        return
    
    if not EMBEDDINGS_AVAILABLE:
        print("Error: sentence-transformers required for semantic search")
        print("Install with: pip install sentence-transformers")
        return
    
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    # Get all documents with embeddings
    c.execute("SELECT id, filename, chunks, embedding FROM documents WHERE embedding IS NOT NULL")
    rows = c.fetchall()
    
    if not rows:
        print("No embedded documents found. Run 'ingest' first.")
        return
    
    # Generate query embedding
    query_embedding = generate_embedding(query)
    if not query_embedding:
        return
    
    # Calculate similarities
    results = []
    for row in rows:
        doc_id, filename, chunks_json, embedding = row
        if embedding:
            sim = cosine_similarity(query_embedding, embedding)
            chunks = json.loads(chunks_json) if chunks_json else []
            results.append((sim, filename, chunks[:3]))  # Top 3 chunks
    
    # Sort by similarity
    results.sort(reverse=True, key=lambda x: x[0])
    
    # Display results
    print(f"\nTop {top_k} results for: '{query}'")
    print("=" * 60)
    
    for i, (score, filename, chunks) in enumerate(results[:top_k], 1):
        print(f"\n{i}. {filename} (score: {score:.3f})")
        print("-" * 40)
        for chunk in chunks[:2]:
            print(f"   {chunk[:200]}...")
    
    conn.close()

def list_documents(pod_name: str):
    """List all documents in pod."""
    pod_path = ensure_pod(pod_name)
    if not pod_path:
        return
    
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    c.execute("SELECT id, filename, file_type, file_hash, created_at FROM documents ORDER BY created_at DESC")
    rows = c.fetchall()
    
    if rows:
        print(f"Documents in '{pod_name}':")
        for row in rows:
            print(f"  [{row[0]}] {row[1]} ({row[2]}) - {row[3]}")
    else:
        print("No documents yet. Run 'ingest' to add documents.")
    
    conn.close()

def main():
    parser = argparse.ArgumentParser(description="Data Pods Ingestion v0.2")
    sub = parser.add_subparsers(dest="cmd")
    
    ingest_p = sub.add_parser("ingest", help="Ingest folder of documents")
    ingest_p.add_argument("pod", help="Pod name")
    ingest_p.add_argument("folder", help="Folder path to ingest")
    ingest_p.add_argument("--no-recursive", action="store_true", help="Don't scan subfolders")
    
    search_p = sub.add_parser("search", help="Semantic search")
    search_p.add_argument("pod", help="Pod name")
    search_p.add_argument("query", help="Search query")
    search_p.add_argument("--top", type=int, default=5, help="Number of results")
    
    list_p = sub.add_parser("list", help="List documents")
    list_p.add_argument("pod", help="Pod name")
    
    sub.add_parser("status", help="Show ingestion status")
    
    args = parser.parse_args()
    
    PODS_DIR.mkdir(parents=True, exist_ok=True)
    
    if args.cmd == "ingest":
        recursive = not args.no_recursive
        ingest_folder(args.pod, args.folder, recursive)
    elif args.cmd == "search":
        search_semantic(args.pod, args.query, args.top)
    elif args.cmd == "list":
        list_documents(args.pod)
    elif args.cmd == "status":
        print("Data Pods Ingestion v0.2")
        print(f"PDF support: {'Yes' if PDF_AVAILABLE else 'No (pip install PyPDF2)'}")
        print(f"DOCX support: {'Yes' if DOCX_AVAILABLE else 'No (pip install python-docx)'}")
        print(f"OCR support: {'Yes' if OCR_AVAILABLE else 'No (pip install pytesseract pillow)'}")
        print(f"Embeddings: {'Yes' if EMBEDDINGS_AVAILABLE else 'No (pip install sentence-transformers)'}")
    else:
        parser.print_help()

if __name__ == "__main__":
    main()


path: scripts/pod.py
content: #!/usr/bin/env python3
"""
Data Pods v0.1 - Modular portable database management
Usage: python pod.py <command> [options]
"""

import sqlite3
import json
import os
import sys
import argparse
from pathlib import Path
from datetime import datetime
import yaml

PODS_DIR = Path.home() / ".openclaw" / "data-pods"

def ensure_dir():
    PODS_DIR.mkdir(parents=True, exist_ok=True)

def create_pod(name: str, pod_type: str = "shared"):
    """Create a new data pod."""
    ensure_dir()
    pod_path = PODS_DIR / name
    if pod_path.exists():
        print(f"Error: Pod '{name}' already exists")
        return False
    
    pod_path.mkdir(parents=True, exist_ok=True)
    
    # Create SQLite database
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    # Default notes table
    c.execute('''CREATE TABLE IF NOT EXISTS notes (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        title TEXT,
        content TEXT,
        tags TEXT,
        created_at TEXT,
        updated_at TEXT
    )''')
    
    # Default embeddings table (placeholder for vector store)
    c.execute('''CREATE TABLE IF NOT EXISTS embeddings (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        note_id INTEGER,
        chunk_text TEXT,
        embedding BLOB,
        FOREIGN KEY(note_id) REFERENCES notes(id)
    )''')
    
    conn.commit()
    conn.close()
    
    # Create metadata
    metadata = {
        "name": name,
        "type": pod_type,
        "created": datetime.now().isoformat(),
        "version": "0.1",
        "tables": ["notes", "embeddings"]
    }
    (pod_path / "metadata.json").write_text(json.dumps(metadata, indent=2))
    
    # Create manifest
    manifest = {
        "name": name,
        "type": pod_type,
        "access": "private",
        "created": metadata["created"],
        "version": "0.1"
    }
    (pod_path / "manifest.yaml").write_text(yaml.dump(manifest))
    
    print(f"✅ Created pod: {name} ({pod_type}) at {pod_path}")
    return True

def list_pods():
    """List all pods."""
    ensure_dir()
    pods = []
    for d in PODS_DIR.iterdir():
        if d.is_dir():
            meta = d / "metadata.json"
            if meta.exists():
                data = json.loads(meta.read_text())
                pods.append(f"- {d.name} ({data.get('type', 'unknown')})")
            else:
                pods.append(f"- {d.name}")
    
    if pods:
        print("📦 Available pods:")
        print("\n".join(pods))
    else:
        print("No pods found. Create one with: pod create <name>")

def add_note(pod_name: str, title: str, content: str, tags: str = ""):
    """Add a note to a pod."""
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return False
    
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    now = datetime.now().isoformat()
    c.execute("INSERT INTO notes (title, content, tags, created_at, updated_at) VALUES (?, ?, ?, ?, ?)",
              (title, content, tags, now, now))
    
    note_id = c.lastrowid
    conn.commit()
    conn.close()
    
    print(f"✅ Added note to '{pod_name}' (ID: {note_id})")
    return True

def query_pod(pod_name: str, text: str = None, sql: str = None):
    """Query a pod."""
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return False
    
    db_path = pod_path / "data.sqlite"
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    
    if sql:
        try:
            c.execute(sql)
            rows = c.fetchall()
            for row in rows:
                print(row)
        except Exception as e:
            print(f"SQL Error: {e}")
    elif text:
        # Simple text search
        c.execute("SELECT id, title, content, tags FROM notes WHERE content LIKE ? OR title LIKE ?", 
                  (f"%{text}%", f"%{text}%"))
        rows = c.fetchall()
        if rows:
            print(f"📄 Found {len(rows)} results for '{text}':")
            for row in rows:
                print(f"  [{row[0]}] {row[1]}: {row[2][:80]}...")
        else:
            print(f"No results for '{text}'")
    else:
        c.execute("SELECT id, title, tags, created_at FROM notes")
        rows = c.fetchall()
        if rows:
            print(f"📄 Notes in '{pod_name}':")
            for row in rows:
                print(f"  [{row[0]}] {row[1]} | {row[2]} | {row[3]}")
        else:
            print("No notes yet.")
    
    conn.close()
    return True

def export_pod(pod_name: str, output_path: str = None):
    """Export pod as zip."""
    import zipfile
    
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return False
    
    if output_path is None:
        output_path = f"{pod_name}.zip"
    
    with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
        for file in pod_path.rglob("*"):
            if file.is_file():
                arcname = file.relative_to(pod_path)
                zipf.write(file, arcname)
    
    print(f"✅ Exported '{pod_name}' to {output_path}")
    return True

def main():
    parser = argparse.ArgumentParser(description="Data Pods v0.1")
    sub = parser.add_subparsers(dest="cmd")
    
    sub.add_parser("list", help="List all pods")
    sub.add_parser("init", help="List all pods (alias)")
    
    create = sub.add_parser("create", help="Create a new pod")
    create.add_argument("name")
    create.add_argument("--type", default="shared", choices=["scholar", "health", "shared", "projects"])
    
    add = sub.add_parser("add", help="Add a note to pod")
    add.add_argument("pod")
    add.add_argument("--title", required=True)
    add.add_argument("--content", required=True)
    add.add_argument("--tags", default="")
    
    query = sub.add_parser("query", help="Query a pod")
    query.add_argument("pod")
    query.add_argument("--text", help="Search text")
    query.add_argument("--sql", help="Raw SQL")
    
    exp = sub.add_parser("export", help="Export pod as zip")
    exp.add_argument("pod")
    exp.add_argument("--output", help="Output path")
    
    args = parser.parse_args()
    
    if args.cmd in ("list", "init"):
        list_pods()
    elif args.cmd == "create":
        create_pod(args.name, args.type)
    elif args.cmd == "add":
        add_note(args.pod, args.title, args.content, args.tags)
    elif args.cmd == "query":
        query_pod(args.pod, args.text, args.sql)
    elif args.cmd == "export":
        export_pod(args.pod, args.output)
    else:
        parser.print_help()

if __name__ == "__main__":
    main()


path: scripts/podsync.py
content: #!/usr/bin/env python3
"""
PodSync v0.1 - Sync and share data pods across devices
Usage: podsync.py <command> [options]
"""

import sqlite3
import json
import os
import sys
import argparse
from pathlib import Path
from datetime import datetime
import zipfile
import shutil
import hashlib

PODS_DIR = Path.home() / ".openclaw" / "data-pods"
SYNC_DIR = Path.home() / ".openclaw" / "sync"

def ensure_dir():
    PODS_DIR.mkdir(parents=True, exist_ok=True)
    SYNC_DIR.mkdir(parents=True, exist_ok=True)

def list_pods():
    """List available pods."""
    ensure_dir()
    pods = []
    for d in PODS_DIR.iterdir():
        if d.is_dir():
            pods.append(d.name)
    return pods

def export_pod(pod_name: str, output_path: str = None):
    """Export pod as a shareable zip."""
    ensure_dir()
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return False
    
    if output_path is None:
        output_path = SYNC_DIR / f"{pod_name}.vpod"
    else:
        output_path = Path(output_path)
    
    output_path.parent.mkdir(parents=True, exist_ok=True)
    
    # Create zip with .vpod extension
    with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
        for file in pod_path.rglob("*"):
            if file.is_file():
                arcname = file.relative_to(pod_path)
                zipf.write(file, arcname)
    
    # Calculate hash for verification
    with open(output_path, 'rb') as f:
        file_hash = hashlib.sha256(f.read()).hexdigest()[:16]
    
    print(f"Exported: {pod_name} -> {output_path}")
    print(f"Hash: {file_hash}")
    print(f"Size: {output_path.stat().st_size / 1024:.1f} KB")
    return True

def import_pod(input_path: str, pod_name: str = None):
    """Import a .vpod file as a pod."""
    ensure_dir()
    input_path = Path(input_path)
    if not input_path.exists():
        print(f"Error: File '{input_path}' not found")
        return False
    
    if pod_name is None:
        pod_name = input_path.stem
    
    pod_path = PODS_DIR / pod_name
    if pod_path.exists():
        print(f"Warning: Pod '{pod_name}' already exists. Use --force to overwrite.")
        return False
    
    # Extract zip
    with zipfile.ZipFile(input_path, 'r') as zipf:
        zipf.extractall(pod_path)
    
    print(f"Imported: {input_path} -> {pod_name}")
    return True

def pack_for_llm(pod_name: str, output_path: str = None):
    """Export pod as a single markdown file for LLM/ChatGPT context."""
    ensure_dir()
    pod_path = PODS_DIR / pod_name
    if not pod_path.exists():
        print(f"Error: Pod '{pod_name}' not found")
        return False
    
    db_path = pod_path / "data.sqlite"
    if not db_path.exists():
        print(f"Error: No database in pod '{pod_name}'")
        return False
    
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("SELECT title, content, tags, created_at FROM notes ORDER BY created_at DESC")
    notes = c.fetchall()
    conn.close()
    
    if output_path is None:
        output_path = SYNC_DIR / f"{pod_name}_for_llm.md"
    else:
        output_path = Path(output_path)
    
    output_path.parent.mkdir(parents=True, exist_ok=True)
    
    # Write as markdown
    with open(output_path, 'w', encoding='utf-8') as f:
        f.write(f"# {pod_name}\n\n")
        f.write(f"Exported: {datetime.now().strftime('%Y-%m-%d %H:%M')}\n\n")
        f.write(f"## Notes\n\n")
        for note in notes:
            title, content, tags, created = note
            f.write(f"### {title}\n")
            f.write(f"*Tags: {tags} | Created: {created}*\n\n")
            f.write(f"{content}\n\n")
            f.write("---\n\n")
    
    print(f"Packed for LLM: {pod_name} -> {output_path}")
    print(f"Size: {output_path.stat().st_size / 1024:.1f} KB")
    print(f"Ready to paste into ChatGPT!")
    return True

def list_exports():
    """List exported pods."""
    ensure_dir()
    files = list(SYNC_DIR.glob("*.vpod")) + list(SYNC_DIR.glob("*.md"))
    if files:
        print("Exported files:")
        for f in files:
            print(f"  - {f.name} ({f.stat().st_size / 1024:.1f} KB)")
    else:
        print("No exports found.")

def sync_status():
    """Show sync status."""
    ensure_dir()
    pods = list_pods()
    exports = list(SYNC_DIR.glob("*.vpod"))
    
    print(f"Pods: {len(pods)}")
    print(f"Exports: {len(exports)}")

def main():
    parser = argparse.ArgumentParser(description="PodSync v0.1 - Sync pods across devices")
    sub = parser.add_subparsers(dest="cmd")
    
    sub.add_parser("list", help="List pods")
    sub.add_parser("status", help="Show sync status")
    sub.add_parser("exports", help="List exported files")
    
    exp = sub.add_parser("export", help="Export pod as .vpod")
    exp.add_argument("pod", help="Pod name")
    exp.add_argument("--output", help="Output path")
    
    imp = sub.add_parser("import", help="Import .vpod file")
    imp.add_argument("file", help="Input file path")
    imp.add_argument("--name", help="Pod name (default: filename)")
    
    pack = sub.add_parser("pack", help="Pack for LLM/ChatGPT as markdown")
    pack.add_argument("pod", help="Pod name")
    pack.add_argument("--output", help="Output markdown path")
    
    args = parser.parse_args()
    
    if args.cmd == "list":
        pods = list_pods()
        if pods:
            print("Available pods:")
            for p in pods:
                print(f"  - {p}")
        else:
            print("No pods found.")
    elif args.cmd == "status":
        sync_status()
    elif args.cmd == "exports":
        list_exports()
    elif args.cmd == "export":
        export_pod(args.pod, args.output)
    elif args.cmd == "import":
        import_pod(args.file, args.name)
    elif args.cmd == "pack":
        pack_for_llm(args.pod, args.output)
    else:
        parser.print_help()

if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)