Skill: teamtelnyx/telnyx-rag (version 1.0.1)

--- name: telnyx-rag description: Semantic search and Q&A over workspace files using Telnyx Storage + AI embeddings. Index your memory, knowledge, and skills for natural language retrieval and AI-powered answers. metadata: {"openclaw":{"emoji":"🧠","requires":{"bins":["python3"],"env":["TELNYX_API_KEY"]},"primaryEnv":"TELNYX_API_KEY"}} --- # Telnyx RAG Memory Semantic search and RAG-powered Q&A over your OpenClaw workspace using Telnyx's native embedding, similarity search, and inference APIs. ## Requirements - **Your own Telnyx API Key** — each user/agent uses their own key - **Python 3.8+** — stdlib only, no external dependencies - Get your API key at [portal.telnyx.com](https://portal.telnyx.com/#/app/api-keys) ## Bucket Naming Convention Use a consistent naming scheme so anyone can adopt this: ``` openclaw-{agent-id} ``` | Agent | Bucket | |-------|--------| | Chief (main) | `openclaw-main` | | Bob the Builder | `openclaw-builder` | | Voice agent | `openclaw-voice` | | Your agent | `openclaw-{your-id}` | **Why?** - **Predictable**: anyone can find any agent's bucket - **Collision-free**: scoped to agent, not person or team - **Discoverable**: `openclaw-*` prefix groups all agent buckets in Telnyx Storage UI ## Quick Start ```bash cd ~/skills/telnyx-rag # Set YOUR Telnyx API key (each user/agent uses their own) echo 'TELNYX_API_KEY=KEY...' > .env # Run setup with validation ./setup.sh --check # Validate requirements first ./setup.sh # Full setup (uses bucket from config.json) # Search your memory ./search.py "What are my preferences?" # Ask questions (full RAG pipeline) ./ask.py "What is the porting process?" ``` ## What It Does - **Indexes** your workspace files (MEMORY.md, memory/*.md, knowledge/, skills/) - **Chunks** large files intelligently (markdown by headers, JSON/Slack by threads) - **Embeds** content automatically using Telnyx AI - **Searches** using natural language queries with retry logic - **Answers questions** using a full RAG pipeline (retrieve → rerank → generate) - **Prioritizes** results from memory/ (your primary context) - **Incremental sync** — only uploads changed files - **Orphan cleanup** — removes deleted files from bucket ## Setup Options ### Option 1: Environment Variable ```bash export TELNYX_API_KEY="KEY..." ./setup.sh ``` ### Option 2: .env File ```bash echo 'TELNYX_API_KEY=KEY...' > .env ./setup.sh ``` ### Validation Mode ```bash ./setup.sh --check # Validate requirements without making changes ``` ### Custom Bucket Name ```bash ./setup.sh my-custom-bucket ``` ## Usage ### Ask Questions (RAG Pipeline) ```bash # Basic question answering ./ask.py "What is Telnyx's porting process?" # Show retrieved context alongside answer ./ask.py "How do I deploy?" --context # Use a different model ./ask.py "Explain voice setup" --model meta-llama/Meta-Llama-3.1-8B-Instruct # More/fewer context chunks ./ask.py "meeting decisions" --num 12 # JSON output for scripting ./ask.py "API usage limits" --json # Search a different bucket ./ask.py "project timeline" --bucket work-memory ``` ### Search Memory ```bash # Basic search with improved error handling ./search.py "What are David's communication preferences?" # Search specific bucket ./search.py "meeting notes" --bucket my-other-bucket # More results with timeout control ./search.py "procedures" --num 10 --timeout 45 # JSON output (for scripts) ./search.py "procedures" --json ``` ### Sync Files (with Chunking) ```bash # Incremental sync with auto-chunking ./sync.py # Override chunk size (tokens) ./sync.py --chunk-size 600 # Quiet mode for cron jobs ./sync.py --quiet # Remove orphaned files (including stale chunks) ./sync.py --prune # Sync + trigger embedding ./sync.py --embed # Check status ./sync.py --status # List indexed files (shows chunks too) ./sync.py --list ``` ### Watch Mode ```bash # Watch for changes and auto-sync with chunking ./sync.py --watch ``` ### Trigger Embedding ```bash # Trigger embedding for current bucket ./embed.sh # OR ./sync.py --embed # Check embedding status ./sync.py --embed-status <task_id> ``` **Why is this needed?** Uploading files to Telnyx Storage doesn't automatically generate embeddings. The embedding process converts your files into searchable vectors. Without this step, `search.py` and `ask.py` won't return results. ## Configuration Edit `config.json` to customize behavior: ```json { "bucket": "openclaw-memory", "region": "us-central-1", "workspace": ".", "patterns": [ "MEMORY.md", "memory/*.md", "knowledge/*.json", "skills/*/SKILL.md" ], "priority_prefixes": ["memory/", "MEMORY.md"], "default_num_docs": 5, "chunk_size": 800, "ask_model": "meta-llama/Meta-Llama-3.1-70B-Instruct", "ask_num_docs": 8, "retrieve_num_docs": 20 } ``` ### Config Fields | Field | Default | Description | |-------|---------|-------------| | `bucket` | `openclaw-{agent-id}` | Telnyx Storage bucket name (see naming convention) | | `region` | `us-central-1` | Storage region | | `workspace` | `.` | Root directory to scan for files | | `patterns` | (see above) | Glob patterns for files to index | | `priority_prefixes` | `["memory/", "MEMORY.md"]` | Sources to rank higher in results | | `exclude` | `["*.tmp", ...]` | Patterns to exclude | | `chunk_size` | `800` | Target tokens per chunk (~4 chars/token) | | `ask_model` | `Meta-Llama-3.1-70B-Instruct` | LLM model for ask.py | | `ask_num_docs` | `8` | Final context chunks for LLM | | `retrieve_num_docs` | `20` | Initial retrieval count (before reranking) | ## How It Works ``` ┌─────────────────┐ ┌──────────────────────────────────┐ │ Your Workspace │ │ Telnyx Cloud │ │ ├── memory/ │ │ │ │ ├── knowledge/ │──┐ │ Storage: your-bucket/ │ │ └── skills/ │ │ │ └── file__chunk-001.md │ └─────────────────┘ │ │ └── file__chunk-002.md │ │ │ │ │ Smart Chunking ◀──┘ │ ▼ embed │ ├── Markdown: split │ Telnyx AI Embeddings │ │ on ## headers │ │ │ ├── JSON/Slack: split │ ▼ │ │ by thread/time │ Similarity Search │ └── Metadata tags │ │ │ └──────────────┼──────────────────┘ │ ask.py Pipeline: │ ┌─────────────────────────────────┐ │ │ 1. Retrieve top-20 chunks ◀────┘ │ │ 2. Rerank (TF-IDF + priority) │ │ 3. Deduplicate adjacent chunks │ │ 4. Build prompt with top-8 │ │ 5. Call Telnyx Inference LLM │ │ 6. Return answer + sources │ └─────────────────────────────────┘ ``` ## Smart Chunking Large files are automatically split into semantic chunks before upload: ### Markdown Files - Split on `##` and `###` headers first - If a section is still too large, split by paragraph boundaries - Each chunk gets a metadata header with source, chunk index, and title ### JSON / Slack Exports - Messages grouped by token budget per chunk - Extracts: channel name, date range, authors - Metadata includes Slack-specific fields ### Chunk Naming Chunks use deterministic filenames: ``` knowledge/meetings.md → knowledge/meetings__chunk-001.md knowledge/meetings__chunk-002.md knowledge/meetings__chunk-003.md ``` ### Chunk Metadata Each chunk includes a YAML-style header: ``` --- source: knowledge/meetings.md chunk: 2/5 title: Q4 Planning Discussion --- (chunk content here) ``` For Slack exports, additional fields: ``` --- source: slack/general.json chunk: 3/12 title: general channel: general date_range: 2024-01-15 to 2024-01-16 authors: alice, bob, charlie --- ``` ### Chunk Lifecycle - When a source file changes, old chunks are deleted and new ones uploaded - Chunk mappings tracked in `.sync-state.json` - `--prune` cleans up orphaned chunks from deleted files ## Reranking (ask.py) The RAG pipeline uses a multi-signal reranking strategy: 1. **Semantic similarity** — Telnyx embedding distance (certainty score) 2. **Keyword overlap** — TF-IDF weighted term matching with the query 3. **Priority boost** — Chunks from `priority_prefixes` sources ranked higher 4. **Deduplication** — Adjacent chunks from the same source with >80% token overlap are merged Initial retrieval fetches `retrieve_num_docs` (default 20), reranking selects the best `ask_num_docs` (default 8) for the LLM prompt. ## New Features (v2) ### Smart Chunking - **Semantic splitting**: Headers for markdown, threads for Slack JSON - **Metadata headers**: Source, chunk index, title in every chunk - **Configurable size**: `--chunk-size` flag or `chunk_size` in config - **Deterministic names**: Reproducible chunk filenames ### RAG Q&A Pipeline (`ask.py`) - **End-to-end**: Query → retrieve → rerank → generate → answer - **Telnyx Inference**: Uses Telnyx LLM API for generation - **Source references**: Every answer includes source file citations - **Context mode**: `--context` shows retrieved chunks - **JSON output**: `--json` for structured responses ### Reranking - **Multi-signal scoring**: Combines embedding similarity + keyword overlap + priority - **Deduplication**: Removes near-identical adjacent chunks - **Configurable**: Retrieve 20, use best 8 (tunable) ### Incremental Sync (v1) - **File hashing**: Tracks SHA-256 hashes in `.sync-state.json` - **Skip unchanged**: Only uploads modified files - **Progress tracking**: Shows progress bars for large syncs ### Smart Cleanup - **`--prune`**: Removes files from bucket that were deleted locally - **Chunk-aware**: Cleans up orphaned chunks too - **State tracking**: Maintains sync history and chunk mappings ### Improved Reliability - **Retry logic**: 3 attempts with exponential backoff - **Better errors**: Parses Telnyx API error responses - **Timeout control**: Configurable request timeouts - **Quiet mode**: `--quiet` flag for cron jobs ## OpenClaw Integration Add to your `TOOLS.md`: ```markdown ## Semantic Memory & Q&A Ask questions about your workspace: \`\`\`bash cd ~/skills/telnyx-rag && ./ask.py "your question" \`\`\` Search memory semantically: \`\`\`bash cd ~/skills/telnyx-rag && ./search.py "your query" \`\`\` ``` ### Automated Sync Add to your heartbeat or cron: ```bash # Quiet sync with orphan cleanup cd ~/skills/telnyx-rag && ./sync.py --quiet --prune # Sync with embedding cd ~/skills/telnyx-rag && ./sync.py --quiet --embed ``` ## Troubleshooting ### Setup Issues **"Python version too old"** - Requires Python 3.8+ - Check: `python3 --version` **"API key test failed"** - Verify key: `echo $TELNYX_API_KEY` - Get new key at [portal.telnyx.com](https://portal.telnyx.com/#/app/api-keys) ### Sync Issues **"Bucket not found"** ```bash ./sync.py --create-bucket ``` **"No results found"** - Wait 1-2 minutes after sync (embeddings take time) - Check files uploaded: `./sync.py --list` - Trigger embedding: `./sync.py --embed` **"Files not syncing"** - Check `.sync-state.json` for corruption - Force re-sync: `rm .sync-state.json && ./sync.py` ### Ask Issues **"LLM generation failed"** - Check API key has inference permissions - Try a different model: `./ask.py "query" --model meta-llama/Meta-Llama-3.1-8B-Instruct` **"No relevant documents found"** - Ensure files are synced and embedded - Try broader query terms ## API Reference ### From Python ```python from ask import ask from search import search_memory # Ask a question (full RAG pipeline) answer = ask("What is the deployment process?") print(answer) # With options answer = ask( "project timeline", num_final=5, model="meta-llama/Meta-Llama-3.1-8B-Instruct", show_context=True, output_json=True, ) print(answer) # Basic search results = search_memory("What do I know about X?", num_docs=5) print(results) ``` ### From Bash ```bash # Ask and capture answer answer=$(./ask.py "What are the API limits?" --json) # Search and capture JSON results=$(./search.py "query" --json) ``` ## Performance Tips 1. **Tune chunk_size** — Smaller chunks (400-600) for precise retrieval, larger (800-1200) for more context 2. **Use `--quiet`** for cron jobs to reduce output 3. **Enable `--prune`** periodically to clean up deleted files 4. **Watch mode** is great for development: `./sync.py --watch` 5. **Batch embedding** by syncing first, then embedding: `./sync.py && ./sync.py --embed` ## Credits Built for [OpenClaw](https://github.com/openclaw/openclaw) using [Telnyx Storage](https://telnyx.com/products/cloud-storage) and AI APIs.

path: ask.py
content: #!/usr/bin/env python3
"""
Telnyx RAG Memory - Ask (RAG Pipeline)

Full RAG pipeline: query -> retrieve top-k chunks -> rerank -> build prompt -> LLM -> answer.

Usage:
  ./ask.py "What is Telnyx's porting process?"
  ./ask.py "How do I set up voice calls?" --num 5
  ./ask.py "meeting notes" --context           # Show retrieved chunks
  ./ask.py "deployment steps" --json           # Structured output
  ./ask.py "API usage" --model meta-llama/Meta-Llama-3.1-8B-Instruct
"""

import os
import sys
import json
import math
import re
import urllib.request
import urllib.error
import time
import argparse
from collections import Counter
from pathlib import Path


def load_config():
    """Load configuration from config.json

    Returns:
        dict: Merged configuration with defaults
    """
    defaults = {
        "bucket": "openclaw-memory",
        "region": "us-central-1",
        "priority_prefixes": ["memory/", "MEMORY.md"],
        "ask_model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
        "ask_num_docs": 8,
        "retrieve_num_docs": 20,
    }
    config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "config.json")
    if os.path.exists(config_path):
        try:
            with open(config_path) as f:
                user_config = json.load(f)
            return {**defaults, **user_config}
        except (json.JSONDecodeError, IOError):
            pass
    return defaults


def load_credentials():
    """Load Telnyx API key from environment or .env file

    Returns:
        str or None: The API key, or None if not found
    """
    [REDACTED_SECRET]"TELNYX_API_KEY")
    if [REDACTED_SECRET] api_key

    env_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".env")
    if os.path.exists(env_file):
        try:
            with open(env_file) as f:
                for line in f:
                    line = line.strip()
                    if line.startswith("TELNYX_API_KEY="):
                        return line.split("=", 1)[1].strip('"').strip("'")
        except IOError:
            pass
    return None


# ---------------------------------------------------------------------------
# Retrieval
# ---------------------------------------------------------------------------

def retrieve_chunks(query, num_docs, bucket, api_key, timeout=30, max_retries=3):
    """Retrieve similar chunks from Telnyx similarity search API

    Args:
        query (str): Search query
        num_docs (int): Number of documents to retrieve
        bucket (str): Bucket name
        api_key (str): Telnyx API key
        timeout (int): Request timeout seconds
        max_retries (int): Max retry attempts

    Returns:
        list[dict]: List of {content, source, certainty} dicts
    """
    url = "https://api.telnyx.com/v2/ai/embeddings/similarity-search"
    payload = json.dumps({
        "bucket_name": bucket,
        "query": query,
        "num_docs": num_docs,
    }).encode()

    headers = {
        "Authorization": "Bearer " + api_key,
        "Content-Type": "application/json",
        "User-Agent": "openclaw-telnyx-rag/2.0",
    }

    for attempt in range(max_retries):
        req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=timeout) as resp:
                data = json.loads(resp.read().decode())
                return _normalize_results(data)
        except urllib.error.HTTPError as e:
            body = e.read().decode("utf-8", errors="ignore")
            if e.code in (401, 403, 404):
                raise RuntimeError("Retrieval failed (HTTP %d): %s" % (e.code, body))
            if attempt < max_retries - 1 and e.code >= 500:
                time.sleep(2 ** attempt)
                continue
            raise RuntimeError("Retrieval failed (HTTP %d): %s" % (e.code, body))
        except (urllib.error.URLError, OSError) as e:
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)
                continue
            raise RuntimeError("Network error during retrieval: %s" % e)

    raise RuntimeError("Max retries exceeded during retrieval")


def _normalize_results(response_data):
    """Normalize Telnyx API response into a flat list of dicts

    Args:
        response_data (dict): Raw API response

    Returns:
        list[dict]: Normalized results
    """
    results = []
    items = []
    if "data" in response_data and isinstance(response_data["data"], list):
        items = response_data["data"]

    for doc in items:
        content = doc.get("document_chunk", doc.get("content", ""))
        meta = doc.get("metadata", {})
        source = meta.get("filename", doc.get("file_name", "unknown"))
        certainty = meta.get("certainty", doc.get("certainty", 0))
        results.append({
            "content": content,
            "source": source,
            "certainty": float(certainty) if certainty else 0.0,
        })
    return results


# ---------------------------------------------------------------------------
# Reranking
# ---------------------------------------------------------------------------

def _tokenize(text):
    """Simple whitespace + punctuation tokenizer

    Args:
        text (str): Input text

    Returns:
        list[str]: Lowercased tokens
    """
    return re.findall(r"[a-z0-9]+", text.lower())


def _idf_scores(docs):
    """Compute inverse-document-frequency for tokens across docs

    Args:
        docs (list[str]): List of document texts

    Returns:
        dict[str, float]: Token -> IDF score
    """
    n = len(docs)
    if n == 0:
        return {}
    df = Counter()
    for doc in docs:
        tokens = set(_tokenize(doc))
        for t in tokens:
            df[t] += 1
    return {t: math.log((n + 1) / (count + 1)) + 1 for t, count in df.items()}


def _keyword_overlap_score(query_tokens, doc_text, idf):
    """Score a document by TF-IDF weighted keyword overlap with the query

    Args:
        query_tokens (list[str]): Tokenized query
        doc_text (str): Document text
        idf (dict): IDF scores

    Returns:
        float: Overlap score
    """
    doc_tokens = Counter(_tokenize(doc_text))
    if not doc_tokens:
        return 0.0
    score = 0.0
    for qt in query_tokens:
        if qt in doc_tokens:
            tf = doc_tokens[qt] / sum(doc_tokens.values())
            score += tf * idf.get(qt, 1.0)
    return score


def _priority_score(source, priority_prefixes):
    """Score a source filename by priority prefix match (lower = better)

    Args:
        source (str): Filename / key
        priority_prefixes (list[str]): Ordered priority prefixes

    Returns:
        int: Priority rank (0 = highest)
    """
    for i, prefix in enumerate(priority_prefixes):
        if source.startswith(prefix):
            return i
    return len(priority_prefixes)


def _is_adjacent_duplicate(a, b):
    """Check if two chunks are near-identical adjacent chunks from same source

    Args:
        a (dict): First chunk
        b (dict): Second chunk

    Returns:
        bool: True if they should be deduplicated
    """
    if a["source"] != b["source"]:
        return False
    # Check for __chunk- pattern
    pat = re.compile(r"^(.+)__chunk-(\d+)\.\w+$")
    ma = pat.match(a["source"])
    mb = pat.match(b["source"])
    if ma and mb and ma.group(1) == mb.group(1):
        if abs(int(ma.group(2)) - int(mb.group(2))) <= 1:
            # Check content similarity via token overlap
            ta = set(_tokenize(a["content"]))
            tb = set(_tokenize(b["content"]))
            if ta and tb:
                overlap = len(ta & tb) / min(len(ta), len(tb))
                return overlap > 0.8
    return False


def rerank(query, chunks, num_final, priority_prefixes):
    """Rerank retrieved chunks using keyword overlap, priority, and dedup

    Args:
        query (str): Original query
        chunks (list[dict]): Retrieved chunks with content/source/certainty
        num_final (int): Number of results to return after reranking
        priority_prefixes (list[str]): Priority source prefixes

    Returns:
        list[dict]: Top-N reranked chunks
    """
    if not chunks:
        return []

    query_tokens = _tokenize(query)
    doc_texts = [c["content"] for c in chunks]
    idf = _idf_scores(doc_texts)

    scored = []
    for c in chunks:
        kw_score = _keyword_overlap_score(query_tokens, c["content"], idf)
        pri = _priority_score(c["source"], priority_prefixes)
        # Combined score: certainty (0-1) + keyword overlap + priority bonus
        combined = c["certainty"] * 2.0 + kw_score * 1.0 + (1.0 / (pri + 1)) * 0.3
        scored.append((combined, c))

    # Sort descending by combined score
    scored.sort(key=lambda x: -x[0])

    # Deduplicate adjacent chunks
    result = []
    for _, chunk in scored:
        is_dup = False
        for existing in result:
            if _is_adjacent_duplicate(chunk, existing):
                is_dup = True
                break
        if not is_dup:
            result.append(chunk)
        if len(result) >= num_final:
            break

    return result


# ---------------------------------------------------------------------------
# LLM Generation
# ---------------------------------------------------------------------------

SYSTEM_PROMPT = (
    "Answer based on the provided context. If the context doesn't contain "
    "enough information, say so. Include references to source files when relevant."
)


def build_prompt(query, chunks):
    """Build the prompt with context chunks for the LLM

    Args:
        query (str): User question
        chunks (list[dict]): Retrieved and reranked chunks

    Returns:
        list[dict]: Messages list for chat completions API
    """
    context_parts = []
    for i, c in enumerate(chunks, 1):
        source = c.get("source", "unknown")
        context_parts.append("[%d] Source: %s\n%s" % (i, source, c["content"]))

    context_block = "\n\n---\n\n".join(context_parts)

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": (
                "Context:\n\n%s\n\n---\n\nQuestion: %s" % (context_block, query)
            ),
        },
    ]
    return messages


def call_llm(messages, model, api_key, timeout=60, max_retries=2):
    """Call Telnyx inference API for chat completion

    Args:
        messages (list[dict]): Chat messages
        model (str): Model identifier
        api_key (str): Telnyx API key
        timeout (int): Request timeout
        max_retries (int): Max retries

    Returns:
        str: Generated answer text
    """
    url = "https://api.telnyx.com/v2/ai/chat/completions"
    payload = json.dumps({
        "model": model,
        "messages": messages,
        "max_tokens": 2048,
        "temperature": 0.3,
    }).encode()

    headers = {
        "Authorization": "Bearer " + api_key,
        "Content-Type": "application/json",
        "User-Agent": "openclaw-telnyx-rag/2.0",
    }

    for attempt in range(max_retries):
        req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
        try:
            with urllib.request.urlopen(req, timeout=timeout) as resp:
                data = json.loads(resp.read().decode())
                choices = data.get("choices", [])
                if choices:
                    return choices[0].get("message", {}).get("content", "")
                return "(No response generated)"
        except urllib.error.HTTPError as e:
            body = e.read().decode("utf-8", errors="ignore")
            if e.code in (401, 403):
                raise RuntimeError("LLM API auth error (HTTP %d): %s" % (e.code, body))
            if attempt < max_retries - 1 and e.code >= 500:
                time.sleep(2 ** attempt)
                continue
            raise RuntimeError("LLM API error (HTTP %d): %s" % (e.code, body))
        except (urllib.error.URLError, OSError) as e:
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)
                continue
            raise RuntimeError("Network error calling LLM: %s" % e)

    raise RuntimeError("Max retries exceeded calling LLM")


# ---------------------------------------------------------------------------
# Main pipeline
# ---------------------------------------------------------------------------

def ask(query, config=None, num_final=None, model=None, bucket=None,
        show_context=False, output_json=False):
    """Full RAG pipeline: retrieve -> rerank -> generate

    Args:
        query (str): User question
        config (dict): Configuration (loaded if None)
        num_final (int): Final number of chunks to use
        model (str): Override LLM model name
        bucket (str): Override bucket name
        show_context (bool): Include retrieved chunks in output
        output_json (bool): Return structured JSON

    Returns:
        str: Answer text or JSON string
    """
    if config is None:
        config = load_config()

    [REDACTED_SECRET])
    if not api_key:
        err = "No Telnyx API key found. Set TELNYX_API_KEY or create .env file."
        if output_json:
            return json.dumps({"error": err})
        return "ERROR: " + err

    bucket = bucket or config.get("bucket", "openclaw-memory")
    model = model or config.get("ask_model", "meta-llama/Meta-Llama-3.1-70B-Instruct")
    retrieve_n = config.get("retrieve_num_docs", 20)
    final_n = num_final or config.get("ask_num_docs", 8)
    priority_prefixes = config.get("priority_prefixes", [])

    # Step 1: Retrieve
    try:
        raw_chunks = retrieve_chunks(query, retrieve_n, bucket, api_key)
    except RuntimeError as e:
        err = "Retrieval failed: %s" % e
        if output_json:
            return json.dumps({"error": err})
        return "ERROR: " + err

    if not raw_chunks:
        msg = "No relevant documents found for your query."
        if output_json:
            return json.dumps({"answer": msg, "sources": [], "chunks_used": 0})
        return msg

    # Step 2: Rerank
    ranked = rerank(query, raw_chunks, final_n, priority_prefixes)

    # Step 3: Build prompt & call LLM
    messages = build_prompt(query, ranked)
    try:
        answer = call_llm(messages, model, api_key)
    except RuntimeError as e:
        err = "LLM generation failed: %s" % e
        if output_json:
            return json.dumps({"error": err})
        return "ERROR: " + err

    # Collect sources
    sources = list(dict.fromkeys(c["source"] for c in ranked))  # unique, ordered

    if output_json:
        result = {
            "answer": answer,
            "model": model,
            "sources": sources,
            "chunks_used": len(ranked),
            "chunks_retrieved": len(raw_chunks),
        }
        if show_context:
            result["context"] = [
                {"source": c["source"], "certainty": c["certainty"],
                 "content": c["content"]}
                for c in ranked
            ]
        return json.dumps(result, indent=2)

    # Text output
    lines = []
    lines.append(answer)
    lines.append("")
    lines.append("📚 Sources: " + ", ".join(sources))

    if show_context:
        lines.append("")
        lines.append("=" * 60)
        lines.append("Retrieved Context Chunks (%d):" % len(ranked))
        lines.append("=" * 60)
        for i, c in enumerate(ranked, 1):
            lines.append("")
            lines.append("[%d] %s (certainty: %.3f)" % (i, c["source"], c["certainty"]))
            lines.append("-" * 40)
            lines.append(c["content"][:1000])
            if len(c["content"]) > 1000:
                lines.append("...[truncated]")

    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(
        description="Ask questions using Telnyx RAG pipeline"
    )
    parser.add_argument("query", nargs="*", help="Question to ask")
    parser.add_argument(
        "--model", "-m", default=None,
        help="LLM model name (default: from config.json)"
    )
    parser.add_argument(
        "--num", "-n", type=int, default=None,
        help="Number of context chunks to use (default: from config.json)"
    )
    parser.add_argument(
        "--bucket", "-b", default=None,
        help="Override bucket name"
    )
    parser.add_argument(
        "--context", "-c", action="store_true",
        help="Show retrieved context chunks alongside answer"
    )
    parser.add_argument(
        "--json", "-j", dest="output_json", action="store_true",
        help="Output structured JSON"
    )

    args = parser.parse_args()

    if not args.query:
        parser.print_help()
        sys.exit(1)

    query = " ".join(args.query)

    if not args.output_json:
        print("\n🤔 Asking: \"%s\"\n" % query)

    result = ask(
        query=query,
        num_final=args.num,
        model=args.model,
        bucket=args.bucket,
        show_context=args.context,
        output_json=args.output_json,
    )

    print(result)


if __name__ == "__main__":
    main()


path: config.json
content: {
  "bucket": "openclaw-main",
  "region": "us-central-1",
  "workspace": ".",
  "patterns": [
    "MEMORY.md",
    "USER.md",
    "SOUL.md",
    "AGENTS.md",
    "TOOLS.md",
    "GUARDRAILS.md",
    "memory/*.md",
    "knowledge/*.json",
    "knowledge/*.md",
    "skills/*/SKILL.md",
    "docs/*.md"
  ],
  "priority_prefixes": ["memory/", "MEMORY.md"],
  "exclude": ["*.tmp", "*~", ".git/*", "__pycache__/*", "node_modules/*"],
  "default_num_docs": 5,
  "chunk_size": 800,
  "ask_model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
  "ask_num_docs": 8,
  "retrieve_num_docs": 20
}


path: diagram.svg
content: <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 800 400" font-family="monospace" font-size="14">
  <defs>
    <marker id="arrow" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
      <path d="M0,0 L0,6 L9,3 z" fill="#666"/>
    </marker>
  </defs>
  
  <!-- Background -->
  <rect width="800" height="400" fill="#1a1a2e"/>
  
  <!-- Your Workspace Box -->
  <rect x="40" y="80" width="200" height="140" fill="none" stroke="#4a9eff" stroke-width="2" stroke-dasharray="5,5" rx="8"/>
  <text x="70" y="110" fill="#fff" font-weight="bold">Your Workspace</text>
  <text x="70" y="140" fill="#888">├── memory/</text>
  <text x="70" y="165" fill="#888">├── knowledge/</text>
  <text x="70" y="190" fill="#888">└── skills/</text>
  
  <!-- Telnyx Cloud Box -->
  <rect x="380" y="40" width="380" height="280" fill="none" stroke="#4a9eff" stroke-width="2" rx="8"/>
  <text x="480" y="75" fill="#fff" font-weight="bold" font-size="16">Telnyx Cloud</text>
  
  <!-- Storage -->
  <text x="420" y="110" fill="#4a9eff">Storage: your-bucket/</text>
  <text x="440" y="135" fill="#888">└── (your files)</text>
  
  <!-- Auto-embed arrow -->
  <line x1="540" y1="150" x2="540" y2="180" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
  <text x="560" y="170" fill="#888" font-size="12">auto-embed</text>
  
  <!-- Telnyx AI Embeddings -->
  <rect x="440" y="185" width="200" height="35" fill="#2d2d44" stroke="#4a9eff" stroke-width="1" rx="4"/>
  <text x="465" y="208" fill="#4a9eff">Telnyx AI Embeddings</text>
  
  <!-- Arrow to Similarity Search -->
  <line x1="540" y1="225" x2="540" y2="255" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
  
  <!-- Similarity Search -->
  <rect x="450" y="260" width="180" height="35" fill="#2d2d44" stroke="#4a9eff" stroke-width="1" rx="4"/>
  <text x="480" y="283" fill="#4a9eff">Similarity Search</text>
  
  <!-- Arrows from workspace to cloud -->
  <line x1="245" y1="150" x2="375" y2="120" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
  
  <!-- Query arrow -->
  <text x="60" y="280" fill="#fff">Query</text>
  <line x1="110" y1="275" x2="445" y2="275" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
  
  <!-- Results arrow -->
  <text x="60" y="320" fill="#fff">Results</text>
  <line x1="445" y1="315" x2="120" y2="315" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
</svg>


path: embed.sh
content: #!/usr/bin/env bash
#
# Telnyx RAG - Trigger Embedding on Bucket
#
# This is now a thin wrapper around sync.py --embed functionality.
# The embedding logic has been moved to sync.py for better integration.
#
# Usage:
#   ./embed.sh                    # Trigger embedding on default bucket
#   ./embed.sh --status <task_id> # Check embedding task status
#

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log_info() { echo -e "${BLUE}ℹ${NC} $1"; }
log_ok()   { echo -e "${GREEN}✓${NC} $1"; }
log_warn() { echo -e "${YELLOW}⚠${NC} $1"; }
log_err()  { echo -e "${RED}✗${NC} $1"; }

main() {
    case "${1:-}" in
        --status|-s)
            if [[ -z "${2:-}" ]]; then
                log_err "Usage: $0 --status <task_id>"
                echo ""
                echo "Check embedding task status using the task ID returned"
                echo "when you triggered embedding."
                exit 1
            fi
            
            log_info "Checking embedding status via sync.py..."
            exec python3 "$SCRIPT_DIR/sync.py" --embed-status "$2"
            ;;
        
        --help|-h)
            echo "Telnyx RAG - Embedding Trigger"
            echo ""
            echo "Usage: $0 [--status <task_id>] [--help]"
            echo ""
            echo "This script triggers embedding on your Telnyx Storage bucket."
            echo "Embedding processes your uploaded files into searchable vectors."
            echo ""
            echo "Options:"
            echo "  --status TASK_ID    Check embedding task status"
            echo "  --help              Show this help"
            echo ""
            echo "Examples:"
            echo "  $0                  # Trigger embedding on default bucket"
            echo "  $0 --status abc123  # Check embedding task status"
            echo ""
            echo "Note: This is now a wrapper around 'sync.py --embed'."
            echo "You can also use the sync.py command directly:"
            echo "  ./sync.py --embed"
            echo "  ./sync.py --embed-status <task_id>"
            ;;
        
        --*)
            log_err "Unknown option: $1"
            echo "Use --help for usage information."
            exit 1
            ;;
        
        *)
            # Default behavior: trigger embedding
            log_info "Triggering embedding via sync.py..."
            exec python3 "$SCRIPT_DIR/sync.py" --embed
            ;;
    esac
}

main "$@"

path: search.py
content: #!/usr/bin/env python3
"""
Telnyx RAG Memory - Semantic Search

Search your memory files using Telnyx AI similarity search.

Usage:
  ./search.py "What are David's preferences?"
  ./search.py "When did we set up voice calls?" --num 3
  ./search.py "project timeline" --json
  ./search.py "meetings" --priority   # Prioritize knowledge/ and skills/
  ./search.py "test" --bucket my-bucket  # Search specific bucket
"""

import os
import sys
import json
import urllib.request
import urllib.error
import urllib.parse
import time
import argparse
from pathlib import Path

# Default configuration
DEFAULT_CONFIG = {
    "bucket": "chief-memory",
    "region": "us-central-1",
    "workspace": "/home/node/clawd",
    "priority_prefixes": ["memory/", "MEMORY.md"],
    "default_num_docs": 5,
}

def load_config():
    """Load configuration from file or defaults"""
    config_path = os.path.join(os.path.dirname(__file__), "config.json")
    if os.path.exists(config_path):
        with open(config_path) as f:
            user_config = json.load(f)
            return {**DEFAULT_CONFIG, **user_config}
    return DEFAULT_CONFIG

def load_credentials():
    """Load Telnyx API key from environment or .env file
    
    Returns:
        str: The Telnyx API key
        
    Raises:
        SystemExit: If no API key found
    """
    # Check environment first
    if os.environ.get("TELNYX_API_KEY"):
        return os.environ["TELNYX_API_KEY"]
    
    # Check for .env file in skill directory
    env_file = os.path.join(os.path.dirname(__file__), ".env")
    if os.path.exists(env_file):
        with open(env_file) as f:
            for line in f:
                line = line.strip()
                if line.startswith("TELNYX_API_KEY="):
                    return line.split("=", 1)[1].strip('"').strip("'")
    
    print("ERROR: No Telnyx API key found", file=sys.stderr)
    print("Set TELNYX_API_KEY environment variable or create .env file with:", file=sys.stderr)
    print("  TELNYX_API_KEY=your-api-key", file=sys.stderr)
    sys.exit(1)

def parse_telnyx_error(error_body):
    """Parse Telnyx API error response for better error messages
    
    Args:
        error_body (str): Raw error response body
        
    Returns:
        str: Human-readable error message
    """
    try:
        error_data = json.loads(error_body)
        if "errors" in error_data and error_data["errors"]:
            error = error_data["errors"][0]
            code = error.get("code", "unknown")
            detail = error.get("detail", "No details provided")
            return f"{code}: {detail}"
        elif "message" in error_data:
            return error_data["message"]
        else:
            return error_body.strip()
    except (json.JSONDecodeError, KeyError):
        return error_body.strip()

def similarity_search_with_retry(query, num_docs=5, bucket_name=None, timeout=30, max_retries=3):
    """Search memory using Telnyx similarity search API with retry logic
    
    Args:
        query (str): Search query text
        num_docs (int): Number of documents to return (default: 5)
        bucket_name (str): Override bucket name from config
        timeout (int): Request timeout in seconds (default: 30)
        max_retries (int): Maximum retry attempts (default: 3)
        
    Returns:
        dict: API response with search results or error info
    """
    config = load_config()
    [REDACTED_SECRET])
    
    # Use provided bucket or fall back to config
    bucket = bucket_name or config["bucket"]
    
    url = "https://api.telnyx.com/v2/ai/embeddings/similarity-search"
    
    payload = json.dumps({
        "bucket_name": bucket,
        "query": query,
        "num_docs": num_docs
    }).encode()
    
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
        "User-Agent": "openclaw-telnyx-rag/1.0"
    }
    
    # Retry with exponential backoff
    for attempt in range(max_retries):
        req = urllib.request.Request(url, data=payload, headers=headers, method='POST')
        
        try:
            with urllib.request.urlopen(req, timeout=timeout) as response:
                response_data = json.loads(response.read().decode())
                # Normalize API response format
                return normalize_api_response(response_data)
                
        except urllib.error.HTTPError as e:
            error_body = e.read().decode('utf-8', errors='ignore')
            
            # Don't retry on certain error codes
            if e.code in [401, 403, 404]:
                return {
                    "error": f"HTTP {e.code}",
                    "details": parse_telnyx_error(error_body)
                }
            
            # Retry on 5xx errors or rate limits
            if attempt < max_retries - 1 and e.code >= 500:
                backoff = 2 ** attempt  # 1, 2, 4 seconds
                time.sleep(backoff)
                continue
            else:
                return {
                    "error": f"HTTP {e.code}",
                    "details": parse_telnyx_error(error_body)
                }
                
        except (urllib.error.URLError, OSError) as e:
            # Network errors - retry with backoff
            if attempt < max_retries - 1:
                backoff = 2 ** attempt
                time.sleep(backoff)
                continue
            else:
                return {"error": f"Network error: {str(e)}"}
        
        except Exception as e:
            return {"error": f"Unexpected error: {str(e)}"}
    
    return {"error": "Max retries exceeded"}

def normalize_api_response(response_data):
    """Normalize API response format for consistent handling
    
    Args:
        response_data (dict): Raw API response
        
    Returns:
        dict: Normalized response with consistent field names
    """
    # Handle both response formats consistently
    if "data" in response_data and isinstance(response_data["data"], list):
        normalized_data = []
        for doc in response_data["data"]:
            # Normalize field names
            normalized_doc = {
                "content": doc.get("document_chunk", doc.get("content", "")),
                "source": doc.get("metadata", {}).get("filename", doc.get("file_name", "unknown")),
                "certainty": doc.get("metadata", {}).get("certainty", doc.get("certainty", 0)),
                "distance": doc.get("distance", 0)
            }
            normalized_data.append(normalized_doc)
        
        return {"data": normalized_data}
    
    return response_data

def prioritize_results(results, priority_prefixes):
    """Re-rank results to prioritize certain directories
    
    Args:
        results (dict): Search results from API
        priority_prefixes (list): List of filename prefixes to prioritize
        
    Returns:
        dict: Results with re-ordered data array
    """
    if not results or "data" not in results:
        return results
    
    def priority_score(doc):
        filename = doc.get("source", "")
        for i, prefix in enumerate(priority_prefixes):
            if filename.startswith(prefix):
                return i  # Lower = higher priority
        return len(priority_prefixes)  # Non-priority files last
    
    def get_certainty(doc):
        return doc.get("certainty", 0)
    
    # Sort by priority first, then by certainty
    results["data"] = sorted(
        results["data"],
        key=lambda d: (priority_score(d), -get_certainty(d))
    )
    
    return results

def format_results(results, max_content_chars=500):
    """Format search results for display
    
    Args:
        results (dict): Search results from API
        max_content_chars (int): Maximum characters to display per result
        
    Returns:
        str: Formatted results string
    """
    if "error" in results:
        return f"❌ Error: {results['error']}\n{results.get('details', '')}"
    
    if "data" not in results or not results["data"]:
        return "No results found."
    
    output = []
    for i, doc in enumerate(results["data"], 1):
        certainty = doc.get("certainty", 0)
        filename = doc.get("source", "unknown")
        content = doc.get("content", "")
        
        # Truncate content if needed
        display_content = content[:max_content_chars]
        
        # Certainty indicator
        if certainty >= 0.9:
            conf = "🟢"
        elif certainty >= 0.85:
            conf = "🟡"
        else:
            conf = "🔴"
        
        output.append(f"--- Result {i} {conf} (certainty: {certainty:.3f}) ---")
        output.append(f"Source: {filename}")
        output.append(f"\n{display_content}")
        if len(content) > max_content_chars:
            output.append("...[truncated]")
        output.append("")
    
    return "\n".join(output)

def search_memory(query, num_docs=5, bucket_name=None, output_json=False, prioritize=True, timeout=30):
    """Main search function
    
    Args:
        query (str): Search query
        num_docs (int): Number of results to return
        bucket_name (str): Override bucket name
        output_json (bool): Return JSON instead of formatted text
        prioritize (bool): Apply priority ranking
        timeout (int): Request timeout in seconds
        
    Returns:
        str: Formatted results or JSON string
    """
    config = load_config()
    
    results = similarity_search_with_retry(
        query, 
        num_docs, 
        bucket_name, 
        timeout
    )
    
    if prioritize and "data" in results:
        results = prioritize_results(results, config.get("priority_prefixes", []))
    
    if output_json:
        return json.dumps(results, indent=2)
    else:
        return format_results(results)

def main():
    parser = argparse.ArgumentParser(description="Search memory with Telnyx RAG")
    parser.add_argument("query", nargs="*", help="Search query")
    parser.add_argument("--num", "-n", type=int, default=5, help="Number of results")
    parser.add_argument("--bucket", "-b", help="Override bucket name from config")
    parser.add_argument("--json", "-j", action="store_true", help="Output as JSON")
    parser.add_argument("--no-priority", action="store_true", help="Don't prioritize results")
    parser.add_argument("--full", "-f", action="store_true", help="Show full content")
    parser.add_argument("--timeout", "-t", type=int, default=30, help="Request timeout in seconds")
    
    args = parser.parse_args()
    
    if not args.query:
        parser.print_help()
        sys.exit(1)
    
    query = " ".join(args.query)
    
    if not args.json:
        bucket_info = f" (bucket: {args.bucket})" if args.bucket else ""
        print(f"\n🔍 Searching: \"{query}\"{bucket_info}\n")
    
    results = search_memory(
        query=query,
        num_docs=args.num,
        bucket_name=args.bucket,
        output_json=args.json,
        prioritize=not args.no_priority,
        timeout=args.timeout
    )
    
    print(results)

if __name__ == "__main__":
    main()

path: setup.sh
content: #!/usr/bin/env bash
#
# Telnyx RAG Memory - Setup Script
#
# Quick setup for semantic search over your OpenClaw workspace.
# Only requirement: TELNYX_API_KEY
#
# Usage: 
#   ./setup.sh [bucket-name]     # Full setup
#   ./setup.sh --check          # Dry-run validation only
#

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BUCKET="${1:-openclaw-memory}"
DRY_RUN=false

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log_info() { echo -e "${BLUE}ℹ${NC} $1"; }
log_ok()   { echo -e "${GREEN}✓${NC} $1"; }
log_warn() { echo -e "${YELLOW}⚠${NC} $1"; }
log_err()  { echo -e "${RED}✗${NC} $1"; }

# Track setup results
CHECKS_PASSED=0
CHECKS_FAILED=0
SETUP_ERRORS=()

check_python() {
    log_info "Checking Python installation..."
    
    if ! command -v python3 &> /dev/null; then
        log_err "Python 3 is required but not installed"
        SETUP_ERRORS+=("Python 3 not found")
        ((CHECKS_FAILED++))
        return 1
    fi
    
    # Check Python version >= 3.8
    python_version=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
    if python3 -c "import sys; sys.exit(0 if sys.version_info >= (3, 8) else 1)" 2>/dev/null; then
        log_ok "Python $python_version found"
        ((CHECKS_PASSED++))
    else
        log_err "Python 3.8+ required, found $python_version"
        SETUP_ERRORS+=("Python version too old: $python_version")
        ((CHECKS_FAILED++))
        return 1
    fi
}

check_syntax() {
    log_info "Validating script syntax..."
    
    # Check Python files
    for py_file in "$SCRIPT_DIR"/*.py; do
        if [[ -f "$py_file" ]]; then
            if python3 -c "import py_compile; py_compile.compile('$py_file', doraise=True)" 2>/dev/null; then
                log_ok "$(basename "$py_file") syntax OK"
            else
                log_err "$(basename "$py_file") has syntax errors"
                SETUP_ERRORS+=("Syntax error in $(basename "$py_file")")
                ((CHECKS_FAILED++))
                return 1
            fi
        fi
    done
    
    # Check bash files
    for sh_file in "$SCRIPT_DIR"/*.sh; do
        if [[ -f "$sh_file" ]]; then
            if bash -n "$sh_file" 2>/dev/null; then
                log_ok "$(basename "$sh_file") syntax OK"
            else
                log_err "$(basename "$sh_file") has syntax errors"
                SETUP_ERRORS+=("Syntax error in $(basename "$sh_file")")
                ((CHECKS_FAILED++))
                return 1
            fi
        fi
    done
    
    ((CHECKS_PASSED++))
}

check_credentials() {
    log_info "Checking Telnyx API credentials..."
    
    # Check for API key
    if [[ -n "${TELNYX_API_KEY:-}" ]]; then
        log_ok "TELNYX_API_KEY found in environment"
    elif [[ -f "$SCRIPT_DIR/.env" ]]; then
        log_ok ".env file found"
        # Source the .env file to get the key
        set -a  # Enable export of all variables
        source "$SCRIPT_DIR/.env"
        set +a  # Disable export
    else
        log_err "No Telnyx API key found"
        echo ""
        echo "   Please set your API key using one of these methods:"
        echo ""
        echo "   Option 1: Environment variable"
        echo "     export TELNYX_API_KEY=\"KEY....\""
        echo ""
        echo "   Option 2: Create .env file in this directory"
        echo "     echo 'TELNYX_API_KEY=KEY....' > $SCRIPT_DIR/.env"
        echo ""
        echo "   Get your API key at: https://portal.telnyx.com/#/app/api-keys"
        echo ""
        SETUP_ERRORS+=("No API key found")
        ((CHECKS_FAILED++))
        return 1
    fi
    
    # Test API key by making a simple request
    if [[ -n "${TELNYX_API_KEY:-}" ]]; then
        log_info "Testing API key validity..."
        
        # Use a simple API call to validate the key
        if test_api_key; then
            log_ok "API key is valid"
            ((CHECKS_PASSED++))
        else
            log_err "API key test failed"
            SETUP_ERRORS+=("Invalid or expired API key")
            ((CHECKS_FAILED++))
            return 1
        fi
    else
        log_warn "Could not load API key for testing"
        ((CHECKS_FAILED++))
        return 1
    fi
}

test_api_key() {
    # Make a simple API call to test the key
    local response
    response=$(curl -s -w "%{http_code}" -o /dev/null \
        -H "Authorization: Bearer $TELNYX_API_KEY" \
        -H "Content-Type: application/json" \
        "https://api.telnyx.com/v2/ai/embeddings/similarity-search" \
        -d '{"bucket_name":"test-nonexistent","query":"test","num_docs":1}' \
        2>/dev/null || echo "000")
    
    # We expect 404 (bucket not found) which means API key works
    # 401/403 would mean bad/expired key
    if [[ "$response" =~ ^[2-4][0-9][0-9]$ ]]; then
        return 0  # API key works (got a valid HTTP response)
    else
        return 1  # Network error or invalid key
    fi
}

check_config() {
    log_info "Validating configuration..."
    
    if [[ ! -f "$SCRIPT_DIR/config.json" ]]; then
        log_err "config.json not found"
        SETUP_ERRORS+=("Missing config.json")
        ((CHECKS_FAILED++))
        return 1
    fi
    
    if python3 -c "import json; json.load(open('$SCRIPT_DIR/config.json'))" 2>/dev/null; then
        log_ok "config.json is valid JSON"
        ((CHECKS_PASSED++))
    else
        log_err "config.json has invalid JSON syntax"
        SETUP_ERRORS+=("Invalid config.json syntax")
        ((CHECKS_FAILED++))
        return 1
    fi
}

make_executable() {
    log_info "Making scripts executable..."
    
    if chmod +x "$SCRIPT_DIR"/*.py "$SCRIPT_DIR"/*.sh 2>/dev/null; then
        log_ok "Scripts marked executable"
        ((CHECKS_PASSED++))
    else
        log_warn "Could not make all scripts executable"
        # Don't fail on this, just warn
    fi
}

update_config() {
    log_info "Updating config.json with bucket name..."
    
    if ! python3 -c "
import json
import sys
try:
    config_path = '$SCRIPT_DIR/config.json'
    with open(config_path) as f:
        config = json.load(f)
    config['bucket'] = '$BUCKET'
    with open(config_path, 'w') as f:
        json.dump(config, f, indent=2)
    print('   ✓ Bucket set to: $BUCKET')
except Exception as e:
    print(f'   ✗ Failed to update config: {e}', file=sys.stderr)
    sys.exit(1)
"; then
        log_err "Failed to update config.json"
        SETUP_ERRORS+=("Could not update config.json")
        ((CHECKS_FAILED++))
        return 1
    fi
    
    ((CHECKS_PASSED++))
}

create_bucket() {
    log_info "Creating bucket..."
    
    if python3 "$SCRIPT_DIR/sync.py" --create-bucket --quiet; then
        log_ok "Bucket created/verified"
        ((CHECKS_PASSED++))
    else
        log_err "Failed to create bucket"
        SETUP_ERRORS+=("Bucket creation failed")
        ((CHECKS_FAILED++))
        return 1
    fi
}

sync_files() {
    log_info "Syncing workspace files..."
    
    if python3 "$SCRIPT_DIR/sync.py" --quiet; then
        log_ok "Files synced successfully"
        ((CHECKS_PASSED++))
    else
        log_err "File sync failed"
        SETUP_ERRORS+=("File sync failed")
        ((CHECKS_FAILED++))
        return 1
    fi
}

trigger_embedding() {
    log_info "Triggering embedding process..."
    
    if python3 "$SCRIPT_DIR/sync.py" --embed --quiet; then
        log_ok "Embedding triggered successfully"
        ((CHECKS_PASSED++))
    else
        log_warn "Embedding trigger failed (you can try manually later)"
        # Don't fail setup for this
    fi
}

show_summary() {
    echo ""
    echo "================================================"
    echo ""
    
    if [[ $CHECKS_FAILED -eq 0 ]]; then
        log_ok "Setup completed successfully! ✨"
        echo ""
        echo "Usage:"
        echo "  Search:  ./search.py \"What do you know about X?\""
        echo "  Sync:    ./sync.py"
        echo "  Status:  ./sync.py --status"
        echo ""
        echo "Note: Embeddings may take 1-2 minutes to process."
        echo "Test with: ./search.py \"test\" after a few minutes."
        echo ""
    else
        log_err "Setup completed with $CHECKS_FAILED errors:"
        echo ""
        for error in "${SETUP_ERRORS[@]}"; do
            echo "  • $error"
        done
        echo ""
        echo "Please fix the above issues and run setup again."
        echo ""
        return 1
    fi
}

show_help() {
    cat << EOF
Telnyx RAG Memory - Setup Script

Usage: $0 [OPTIONS] [bucket-name]

OPTIONS:
  --check     Validate requirements without making changes
  --help      Show this help message

ARGUMENTS:
  bucket-name Custom bucket name (default: openclaw-memory)

EXAMPLES:
  $0                    # Full setup with default bucket
  $0 my-bucket          # Full setup with custom bucket
  $0 --check            # Validate requirements only

REQUIREMENTS:
  - Python 3.8+
  - TELNYX_API_KEY environment variable or .env file
  - Network access to Telnyx APIs

Get your API key at: https://portal.telnyx.com/#/app/api-keys
EOF
}

main() {
    # Parse arguments
    case "${1:-}" in
        --check)
            DRY_RUN=true
            shift
            BUCKET="${1:-openclaw-memory}"
            ;;
        --help|-h)
            show_help
            exit 0
            ;;
        --*)
            log_err "Unknown option: $1"
            echo "Use --help for usage information."
            exit 1
            ;;
        *)
            # Positional argument is bucket name
            BUCKET="${1:-openclaw-memory}"
            ;;
    esac
    
    echo ""
    echo "🧠 Telnyx RAG Memory Setup"
    echo "=========================="
    echo ""
    echo "Bucket: $BUCKET"
    if [[ "$DRY_RUN" == "true" ]]; then
        echo "Mode: Validation Only (--check)"
    fi
    echo ""
    
    # Run checks
    check_python
    check_syntax
    check_credentials
    check_config
    
    if [[ "$DRY_RUN" == "true" ]]; then
        echo ""
        log_info "Dry-run complete. No changes were made."
        show_summary
        return $?
    fi
    
    # Proceed with setup if checks pass
    if [[ $CHECKS_FAILED -eq 0 ]]; then
        make_executable
        update_config
        create_bucket
        sync_files
        trigger_embedding
    fi
    
    show_summary
}

main "$@"

path: sync.py
content: #!/usr/bin/env python3
"""
Telnyx RAG Memory - Sync Script

Syncs workspace files to Telnyx Storage for semantic search.
Files are auto-embedded by Telnyx AI when embeddings are enabled on the bucket.

Supports smart chunking for large files (markdown, JSON/Slack exports).

Usage:
  ./sync.py                 # Sync all configured files
  ./sync.py --watch         # Watch for changes and sync in real-time
  ./sync.py --list          # List files in bucket
  ./sync.py --create-bucket # Create the memory bucket
  ./sync.py --status        # Check bucket and embedding status
  ./sync.py --prune         # Remove orphaned files from bucket
  ./sync.py --quiet         # Only show errors (good for cron)
  ./sync.py --embed         # Trigger embedding after sync
  ./sync.py --embed-status <task_id>  # Check embedding task status
  ./sync.py --chunk-size 600          # Override chunk size (tokens)
"""

import os
import sys
import json
import hashlib
import hmac
import datetime
import urllib.request
import urllib.error
import urllib.parse
import time
import re
import argparse
from pathlib import Path

# Default configuration
DEFAULT_CONFIG = {
    "bucket": "chief-memory",
    "region": "us-central-1",
    "workspace": "/home/node/clawd",
    "patterns": [
        "MEMORY.md",
        "USER.md",
        "SOUL.md",
        "AGENTS.md",
        "TOOLS.md",
        "GUARDRAILS.md",
        "memory/*.md",
        "knowledge/*.json",
        "knowledge/*.md",
        "skills/*/SKILL.md",
        "docs/*.md",
    ],
    "exclude": [
        "*.tmp",
        "*~",
        ".git/*",
        "__pycache__/*",
        "node_modules/*",
    ],
    "priority_prefixes": ["memory/", "MEMORY.md"],
    "chunk_size": 800,
}

# ---------------------------------------------------------------------------
# Chunking helpers
# ---------------------------------------------------------------------------

def _estimate_tokens(text):
    """Estimate token count from character count (approx 1 token per 4 chars)

    Args:
        text (str): Input text

    Returns:
        int: Estimated token count
    """
    return max(1, len(text) // 4)


def _make_metadata_header(source, chunk_index, total_chunks, title=""):
    """Build a YAML-style metadata header for a chunk

    Args:
        source (str): Original file path
        chunk_index (int): 1-based chunk index
        total_chunks (int): Total chunks for this file
        title (str): Section title

    Returns:
        str: Metadata header block
    """
    lines = ["---"]
    lines.append("source: %s" % source)
    lines.append("chunk: %d/%d" % (chunk_index, total_chunks))
    if title:
        lines.append("title: %s" % title)
    lines.append("---")
    return "\n".join(lines)


def _chunk_key(original_key, index, ext=".md"):
    """Generate deterministic chunk filename

    Args:
        original_key (str): Original S3 key / relative path
        index (int): 1-based chunk index
        ext (str): File extension for chunks

    Returns:
        str: Chunk key like  'path/file__chunk-001.md'
    """
    base, _ = os.path.splitext(original_key)
    return "%s__chunk-%03d%s" % (base, index, ext)


def _extract_heading(text):
    """Extract first markdown heading from text

    Args:
        text (str): Markdown text

    Returns:
        str: Heading text or empty string
    """
    for line in text.split("\n"):
        line = line.strip()
        if line.startswith("#"):
            return line.lstrip("#").strip()
    return ""


def _split_by_paragraphs(text, max_tokens):
    """Split text into chunks by paragraph boundaries respecting token limit

    Args:
        text (str): Input text
        max_tokens (int): Max tokens per chunk

    Returns:
        list[str]: List of text chunks
    """
    paragraphs = re.split(r"\n\s*\n", text)
    chunks = []
    current = []
    current_tokens = 0

    for para in paragraphs:
        para = para.strip()
        if not para:
            continue
        pt = _estimate_tokens(para)
        if current and (current_tokens + pt) > max_tokens:
            chunks.append("\n\n".join(current))
            current = [para]
            current_tokens = pt
        else:
            current.append(para)
            current_tokens += pt

    if current:
        chunks.append("\n\n".join(current))

    return chunks if chunks else [text]


def chunk_markdown(text, source, max_tokens=800):
    """Split markdown into semantic chunks by headers then paragraphs

    Args:
        text (str): Markdown content
        source (str): Original file path for metadata
        max_tokens (int): Target max tokens per chunk

    Returns:
        list[tuple[str, str]]: List of (chunk_content_with_metadata, title)
    """
    if _estimate_tokens(text) <= max_tokens:
        title = _extract_heading(text) or os.path.basename(source)
        header = _make_metadata_header(source, 1, 1, title)
        return [(header + "\n\n" + text, title)]

    # Split on level-2 and level-3 headers
    sections = re.split(r"(?m)^(#{2,3}\s+.+)$", text)

    # Reassemble into (heading, body) pairs
    parts = []
    current_heading = ""
    current_body_parts = []

    i = 0
    while i < len(sections):
        section = sections[i]
        if re.match(r"^#{2,3}\s+", section.strip()):
            # Save previous
            if current_body_parts:
                body = "\n".join(current_body_parts).strip()
                if body:
                    parts.append((current_heading, body))
            current_heading = section.strip().lstrip("#").strip()
            current_body_parts = []
        else:
            current_body_parts.append(section)
        i += 1

    # Save last section
    if current_body_parts:
        body = "\n".join(current_body_parts).strip()
        if body:
            parts.append((current_heading, body))

    if not parts:
        parts = [("", text)]

    # Sub-split large sections by paragraph
    final_parts = []
    for heading, body in parts:
        if _estimate_tokens(body) > max_tokens:
            sub_chunks = _split_by_paragraphs(body, max_tokens)
            for j, sc in enumerate(sub_chunks):
                sub_title = heading
                if len(sub_chunks) > 1:
                    sub_title = "%s (part %d)" % (heading, j + 1) if heading else ""
                final_parts.append((sub_title, sc))
        else:
            final_parts.append((heading, body))

    total = len(final_parts)
    results = []
    for idx, (title, body) in enumerate(final_parts, 1):
        display_title = title or _extract_heading(body) or os.path.basename(source)
        header = _make_metadata_header(source, idx, total, display_title)
        results.append((header + "\n\n" + body, display_title))

    return results


def chunk_slack_json(text, source, max_tokens=800):
    """Split Slack-style JSON exports into chunks by time window / thread

    Args:
        text (str): JSON content (array of message objects)
        source (str): Original file path
        max_tokens (int): Target max tokens per chunk

    Returns:
        list[tuple[str, str]]: List of (chunk_content_with_metadata, title)
    """
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        # Fall back to markdown chunking for invalid JSON
        return chunk_markdown(text, source, max_tokens)

    # Handle both array of messages and object with messages key
    messages = []
    channel_name = ""
    if isinstance(data, list):
        messages = data
        # Try to extract channel from filename
        channel_name = os.path.basename(source).replace(".json", "")
    elif isinstance(data, dict):
        messages = data.get("messages", data.get("data", []))
        channel_name = data.get("channel", data.get("name", ""))
        if not isinstance(messages, list):
            return chunk_markdown(text, source, max_tokens)

    if not messages:
        return chunk_markdown(text, source, max_tokens)

    # Group messages into chunks by token budget
    chunks_raw = []
    current_msgs = []
    current_tokens = 0
    current_authors = set()
    current_dates = []

    for msg in messages:
        if not isinstance(msg, dict):
            continue
        # Extract text content
        msg_text = msg.get("text", msg.get("content", ""))
        user = msg.get("user", msg.get("username", msg.get("author", "unknown")))
        ts = msg.get("ts", msg.get("timestamp", msg.get("date", "")))

        formatted = "[%s] %s: %s" % (str(ts)[:10] if ts else "?", user, msg_text)
        mt = _estimate_tokens(formatted)

        if current_msgs and (current_tokens + mt) > max_tokens:
            chunks_raw.append((current_msgs[:], current_authors.copy(), current_dates[:]))
            current_msgs = [formatted]
            current_tokens = mt
            current_authors = {user}
            current_dates = [str(ts)[:10] if ts else ""]
        else:
            current_msgs.append(formatted)
            current_tokens += mt
            current_authors.add(user)
            if ts:
                current_dates.append(str(ts)[:10])

    if current_msgs:
        chunks_raw.append((current_msgs[:], current_authors.copy(), current_dates[:]))

    total = len(chunks_raw)
    results = []
    for idx, (msgs, authors, dates) in enumerate(chunks_raw, 1):
        date_range = ""
        clean_dates = sorted(set(d for d in dates if d))
        if clean_dates:
            if len(clean_dates) == 1:
                date_range = clean_dates[0]
            else:
                date_range = "%s to %s" % (clean_dates[0], clean_dates[-1])

        title = channel_name or os.path.basename(source)
        meta_lines = ["---"]
        meta_lines.append("source: %s" % source)
        meta_lines.append("chunk: %d/%d" % (idx, total))
        meta_lines.append("title: %s" % title)
        if channel_name:
            meta_lines.append("channel: %s" % channel_name)
        if date_range:
            meta_lines.append("date_range: %s" % date_range)
        if authors:
            meta_lines.append("authors: %s" % ", ".join(sorted(authors)))
        meta_lines.append("---")
        header = "\n".join(meta_lines)

        body = "\n".join(msgs)
        results.append((header + "\n\n" + body, title))

    return results if results else chunk_markdown(text, source, max_tokens)


def chunk_file(filepath, s3_key, max_tokens=800):
    """Chunk a file based on its type

    Args:
        filepath (Path): Local file path
        s3_key (str): Relative path / S3 key
        max_tokens (int): Target max tokens per chunk

    Returns:
        list[tuple[str, str, str]]: List of (chunk_key, content, title)
    """
    try:
        with open(filepath, "r", encoding="utf-8", errors="replace") as f:
            text = f.read()
    except IOError:
        return []

    if not text.strip():
        return []

    # Small files don't need chunking - return as-is without metadata wrapping
    if _estimate_tokens(text) <= max_tokens:
        return [(s3_key, text, "")]

    # Route by file type
    ext = os.path.splitext(str(filepath))[1].lower()
    if ext == ".json":
        chunks = chunk_slack_json(text, s3_key, max_tokens)
    else:
        chunks = chunk_markdown(text, s3_key, max_tokens)

    # Generate chunk keys
    result = []
    for idx, (content, title) in enumerate(chunks, 1):
        ck = _chunk_key(s3_key, idx)
        result.append((ck, content, title))

    return result


# ---------------------------------------------------------------------------
# Sync state
# ---------------------------------------------------------------------------

class SyncState:
    """Manages sync state tracking with file hashes and chunk mappings"""

    def __init__(self, state_file=".sync-state.json"):
        self.state_file = state_file
        self.state = self._load_state()

    def _load_state(self):
        """Load sync state from disk"""
        if os.path.exists(self.state_file):
            try:
                with open(self.state_file, "r") as f:
                    return json.load(f)
            except (json.JSONDecodeError, IOError):
                pass
        return {"file_hashes": {}, "chunk_map": {}, "last_sync": None}

    def save_state(self):
        """Save sync state to disk"""
        self.state["last_sync"] = datetime.datetime.utcnow().isoformat()
        try:
            with open(self.state_file, "w") as f:
                json.dump(self.state, f, indent=2)
        except IOError as e:
            print("Warning: Could not save sync state: %s" % e, file=sys.stderr)

    def get_file_hash(self, filepath):
        """Get stored hash for a file"""
        return self.state["file_hashes"].get(str(filepath))

    def set_file_hash(self, filepath, file_hash):
        """Store hash for a file"""
        self.state["file_hashes"][str(filepath)] = file_hash

    def remove_file(self, filepath):
        """Remove file from sync state"""
        self.state["file_hashes"].pop(str(filepath), None)

    def get_tracked_files(self):
        """Get set of all tracked files (including chunk keys)"""
        tracked = set(self.state["file_hashes"].keys())
        for chunks in self.state.get("chunk_map", {}).values():
            tracked.update(chunks)
        return tracked

    def get_chunk_keys(self, source_key):
        """Get list of chunk keys for a source file

        Args:
            source_key (str): Original file S3 key

        Returns:
            list[str]: List of chunk keys
        """
        return self.state.get("chunk_map", {}).get(source_key, [])

    def set_chunk_keys(self, source_key, chunk_keys):
        """Store chunk key mapping for a source file

        Args:
            source_key (str): Original file S3 key
            chunk_keys (list[str]): Chunk keys produced
        """
        if "chunk_map" not in self.state:
            self.state["chunk_map"] = {}
        self.state["chunk_map"][source_key] = chunk_keys

    def remove_chunks(self, source_key):
        """Remove chunk mapping for a source file

        Args:
            source_key (str): Original file S3 key

        Returns:
            list[str]: Old chunk keys that were tracked
        """
        return self.state.get("chunk_map", {}).pop(source_key, [])

    def get_all_chunk_keys(self):
        """Get all chunk keys across all source files

        Returns:
            set[str]: All chunk keys
        """
        result = set()
        for chunks in self.state.get("chunk_map", {}).values():
            result.update(chunks)
        return result


# ---------------------------------------------------------------------------
# S3 Client
# ---------------------------------------------------------------------------

class TelnyxS3Client:
    """Clean S3 client for Telnyx Cloud Storage using AWS SigV4"""

    def __init__(self, api_key, region):
        self.[REDACTED_SECRET]
        self.region = region
        self.secret_key = "placeholder"  # Telnyx uses API key as access key

    def _sign(self, key, msg):
        """AWS SigV4 signing helper"""
        return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

    def _get_signature_key(self, date_stamp):
        """Generate AWS SigV4 signing key"""
        k_date = self._sign(("AWS4" + self.secret_key).encode("utf-8"), date_stamp)
        k_region = self._sign(k_date, self.region)
        k_service = self._sign(k_region, "s3")
        k_signing = self._sign(k_service, "aws4_request")
        return k_signing

    def _make_request(self, method, bucket, key, payload=b"", content_type=None, extra_headers=None):
        """Make authenticated S3 request"""
        host = (
            "%s.%s.telnyxcloudstorage.com" % (bucket, self.region)
            if bucket
            else "%s.telnyxcloudstorage.com" % self.region
        )
        path = "/%s" % key if key else "/"

        t = datetime.datetime.utcnow()
        amz_date = t.strftime("%Y%m%dT%H%M%SZ")
        date_stamp = t.strftime("%Y%m%d")

        payload_hash = hashlib.sha256(payload).hexdigest()

        headers = {
            "host": host,
            "x-amz-content-sha256": payload_hash,
            "x-amz-date": amz_date,
        }

        if content_type:
            headers["content-type"] = content_type
        if payload:
            headers["content-length"] = str(len(payload))
        if extra_headers:
            headers.update(extra_headers)

        canonical_headers = "".join("%s:%s\n" % (k, v) for k, v in sorted(headers.items()))
        signed_headers = ";".join(sorted(headers.keys()))

        canonical_request = "%s\n%s\n\n%s\n%s\n%s" % (
            method, path, canonical_headers, signed_headers, payload_hash
        )

        algorithm = "AWS4-HMAC-SHA256"
        credential_scope = "%s/%s/s3/aws4_request" % (date_stamp, self.region)
        string_to_sign = "%s\n%s\n%s\n%s" % (
            algorithm,
            amz_date,
            credential_scope,
            hashlib.sha256(canonical_request.encode()).hexdigest(),
        )

        signing_key = self._get_signature_key(date_stamp)
        signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()

        auth = "%s Credential=%s/%s, SignedHeaders=%s, Signature=%s" % (
            algorithm, self.api_key, credential_scope, signed_headers, signature
        )

        request_headers = {
            "x-amz-date": amz_date,
            "x-amz-content-sha256": payload_hash,
            "Authorization": auth,
        }
        if content_type:
            request_headers["Content-Type"] = content_type
        if payload:
            request_headers["Content-Length"] = str(len(payload))
        if extra_headers:
            request_headers.update(extra_headers)

        url = "https://%s%s" % (host, path)
        req = urllib.request.Request(
            url, data=payload if payload else None, headers=request_headers, method=method
        )

        try:
            with urllib.request.urlopen(req, timeout=30) as response:
                return response.status, response.read().decode("utf-8", errors="ignore")
        except urllib.error.HTTPError as e:
            return e.code, e.read().decode("utf-8", errors="ignore")
        except Exception as e:
            return 0, str(e)

    def put_object(self, bucket, key, data, content_type=None):
        """Upload object to bucket"""
        if isinstance(data, str):
            data = data.encode("utf-8")

        if not content_type:
            if key.endswith(".md"):
                content_type = "text/markdown"
            elif key.endswith(".json"):
                content_type = "application/json"
            else:
                content_type = "text/plain"

        status, _ = self._make_request("PUT", bucket, key, data, content_type)
        return status in [200, 204]

    def head_object(self, bucket, key):
        """Check if object exists and get metadata"""
        status, _ = self._make_request("HEAD", bucket, key)
        return status == 200

    def list_objects(self, bucket, prefix=""):
        """List objects in bucket"""
        query = "?prefix=%s" % urllib.parse.quote(prefix) if prefix else ""
        status, body = self._make_request("GET", bucket, "" + query)

        if status == 200:
            files = []
            for match in re.finditer(r"<Key>([^<]+)</Key>", body):
                files.append(match.group(1))
            return files
        return []

    def delete_object(self, bucket, key):
        """Delete object from bucket"""
        status, _ = self._make_request("DELETE", bucket, key)
        return status in [200, 204]

    def create_bucket(self, bucket):
        """Create bucket in region"""
        bucket_config = (
            '<?xml version="1.0" encoding="UTF-8"?>\n'
            '<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">\n'
            "  <LocationConstraint>%s</LocationConstraint>\n"
            "</CreateBucketConfiguration>" % self.region
        )

        host = "%s.telnyxcloudstorage.com" % self.region
        path = "/%s" % bucket

        t = datetime.datetime.utcnow()
        amz_date = t.strftime("%Y%m%dT%H%M%SZ")
        date_stamp = t.strftime("%Y%m%d")

        payload = bucket_config.encode()
        payload_hash = hashlib.sha256(payload).hexdigest()

        headers = {
            "host": host,
            "x-amz-content-sha256": payload_hash,
            "x-amz-date": amz_date,
            "content-type": "application/xml",
            "content-length": str(len(payload)),
        }

        canonical_headers = "".join("%s:%s\n" % (k, v) for k, v in sorted(headers.items()))
        signed_headers = ";".join(sorted(headers.keys()))

        canonical_request = "PUT\n%s\n\n%s\n%s\n%s" % (
            path, canonical_headers, signed_headers, payload_hash
        )

        algorithm = "AWS4-HMAC-SHA256"
        credential_scope = "%s/%s/s3/aws4_request" % (date_stamp, self.region)
        string_to_sign = "%s\n%s\n%s\n%s" % (
            algorithm,
            amz_date,
            credential_scope,
            hashlib.sha256(canonical_request.encode()).hexdigest(),
        )

        signing_key = self._get_signature_key(date_stamp)
        signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()

        auth = "%s Credential=%s/%s, SignedHeaders=%s, Signature=%s" % (
            algorithm, self.api_key, credential_scope, signed_headers, signature
        )

        request_headers = {
            "x-amz-date": amz_date,
            "x-amz-content-sha256": payload_hash,
            "Authorization": auth,
            "Content-Type": "application/xml",
            "Content-Length": str(len(payload)),
        }

        url = "https://%s%s" % (host, path)
        req = urllib.request.Request(url, data=payload, headers=request_headers, method="PUT")

        try:
            with urllib.request.urlopen(req, timeout=30) as response:
                status = response.status
        except urllib.error.HTTPError as e:
            status = e.code
        except Exception:
            status = 0

        return status in [200, 204] or status == 409  # 409 = already exists


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def load_config():
    """Load configuration from file or defaults"""
    config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "config.json")
    if os.path.exists(config_path):
        try:
            with open(config_path) as f:
                user_config = json.load(f)
                return {**DEFAULT_CONFIG, **user_config}
        except (json.JSONDecodeError, IOError):
            pass
    return DEFAULT_CONFIG


def load_credentials():
    """Load Telnyx API key from environment or .env file"""
    [REDACTED_SECRET]"TELNYX_API_KEY")
    if [REDACTED_SECRET] api_key

    env_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".env")
    if os.path.exists(env_file):
        try:
            with open(env_file) as f:
                for line in f:
                    line = line.strip()
                    if line.startswith("TELNYX_API_KEY="):
                        return line.split("=", 1)[1].strip('"').strip("'")
        except IOError:
            pass
    return None


def calculate_file_hash(filepath):
    """Calculate SHA-256 hash of file contents"""
    hash_sha256 = hashlib.sha256()
    try:
        with open(filepath, "rb") as f:
            for chunk in iter(lambda: f.read(4096), b""):
                hash_sha256.update(chunk)
        return hash_sha256.hexdigest()
    except IOError:
        return None


def get_files_to_sync(config):
    """Get list of files to sync based on patterns"""
    workspace = Path(config["workspace"])
    files = []

    for pattern in config["patterns"]:
        for path in workspace.glob(pattern):
            if path.is_file():
                rel_path = path.relative_to(workspace)

                excluded = False
                for excl in config.get("exclude", []):
                    if path.match(excl):
                        excluded = True
                        break

                if not excluded:
                    files.append((path, str(rel_path)))

    return files


def make_api_request(method, endpoint, payload=None):
    """Make Telnyx API request (for embeddings)"""
    [REDACTED_SECRET])
    if not api_key:
        raise ValueError("No API key available")

    url = "https://api.telnyx.com/v2/%s" % endpoint
    data = json.dumps(payload).encode() if payload else None

    headers = {
        "Authorization": "Bearer %s" % api_key,
        "Content-Type": "application/json",
    }

    req = urllib.request.Request(url, data=data, headers=headers, method=method)

    try:
        with urllib.request.urlopen(req, timeout=60) as response:
            return response.status, json.loads(response.read().decode())
    except urllib.error.HTTPError as e:
        return e.code, e.read().decode("utf-8", errors="ignore")
    except Exception as e:
        return 0, str(e)


def show_progress(current, total, prefix="Progress"):
    """Display a simple progress bar"""
    if total == 0:
        return

    percent = int((current / total) * 100)
    filled = int((current / total) * 40)
    bar = "\u2588" * filled + "\u2591" * (40 - filled)
    print("\r%s: [%s] %d%% (%d/%d)" % (prefix, bar, percent, current, total), end="", flush=True)


# ---------------------------------------------------------------------------
# Sync logic
# ---------------------------------------------------------------------------

def _delete_old_chunks(client, bucket, state, s3_key, quiet=False):
    """Delete previously uploaded chunks for a source file

    Args:
        client (TelnyxS3Client): S3 client
        bucket (str): Bucket name
        state (SyncState): Sync state
        s3_key (str): Source file key
        quiet (bool): Suppress output

    Returns:
        int: Number of chunks deleted
    """
    old_chunks = state.get_chunk_keys(s3_key)
    deleted = 0
    for ck in old_chunks:
        if client.delete_object(bucket, ck):
            deleted += 1
            if not quiet:
                print("  🗑️  %s" % ck)
    return deleted


def sync_files(config, quiet=False, show_progress_bar=True, chunk_size_override=None):
    """Sync all configured files to the bucket with incremental updates and chunking

    Args:
        config (dict): Configuration
        quiet (bool): Suppress output
        show_progress_bar (bool): Show progress
        chunk_size_override (int or None): Override chunk size in tokens

    Returns:
        tuple[int, int]: (success_count, failed_count)
    """
    [REDACTED_SECRET])
    if not api_key:
        if not quiet:
            print("ERROR: No Telnyx API key found.")
            print("Set TELNYX_API_KEY environment variable or create .env file")
        return 0, 0

    client = TelnyxS3Client(api_key, config["region"])
    state = SyncState()
    files = get_files_to_sync(config)
    max_tokens = chunk_size_override or config.get("chunk_size", 800)

    if not quiet:
        print("\n\U0001f504 Syncing %d files to %s (chunk size: %d tokens)" % (
            len(files), config["bucket"], max_tokens))
        if not files:
            print("No files match the configured patterns.")
            return 0, 0
        print()

    success = 0
    failed = 0
    skipped = 0

    for i, (local_path, s3_key) in enumerate(files, 1):
        if show_progress_bar and not quiet:
            show_progress(i - 1, len(files), "Checking")

        current_hash = calculate_file_hash(local_path)
        if not current_hash:
            failed += 1
            if not quiet:
                print("\n  \u2717 %s (could not read)" % s3_key)
            continue

        stored_hash = state.get_file_hash(s3_key)
        if stored_hash == current_hash:
            skipped += 1
            continue

        # File changed or new — chunk and upload
        chunks = chunk_file(local_path, s3_key, max_tokens)
        if not chunks:
            failed += 1
            if not quiet:
                print("\n  \u2717 %s (empty or unreadable)" % s3_key)
            continue

        # Delete old chunks first
        _delete_old_chunks(client, config["bucket"], state, s3_key, quiet)

        # Upload chunks (or the single file if no chunking needed)
        chunk_keys = []
        all_ok = True
        for ck, content, title in chunks:
            if client.put_object(config["bucket"], ck, content):
                chunk_keys.append(ck)
            else:
                all_ok = False
                if not quiet:
                    print("\n  \u2717 %s" % ck)

        if all_ok:
            success += 1
            state.set_file_hash(s3_key, current_hash)
            if len(chunks) > 1:
                state.set_chunk_keys(s3_key, chunk_keys)
                if not quiet:
                    print("\n  \u2713 %s (%d chunks)" % (s3_key, len(chunks)))
            else:
                # Single file, no chunk mapping needed — clear any old mapping
                state.remove_chunks(s3_key)
                if not quiet:
                    print("\n  \u2713 %s" % s3_key)
        else:
            failed += 1

    if show_progress_bar and not quiet:
        print()  # Final newline

    state.save_state()

    if not quiet:
        print("\n\u2705 Synced: %d | Skipped: %d | Failed: %d" % (success, skipped, failed))

    return success, failed


def prune_orphaned_files(config, quiet=False):
    """Remove files from bucket that no longer exist locally"""
    [REDACTED_SECRET])
    if not api_key:
        if not quiet:
            print("ERROR: No API key available for pruning")
        return 0

    client = TelnyxS3Client(api_key, config["region"])
    state = SyncState()

    local_files = set(s3_key for _, s3_key in get_files_to_sync(config))

    bucket_files = set(client.list_objects(config["bucket"]))
    tracked_files = state.get_tracked_files()

    orphaned = (bucket_files & tracked_files) - local_files
    # Also include chunk keys whose source no longer exists
    all_chunk_keys = state.get_all_chunk_keys()
    orphaned = orphaned | (all_chunk_keys - set().union(
        *(set(state.get_chunk_keys(k)) for k in local_files if state.get_chunk_keys(k))
    ) if all_chunk_keys else set())

    if not orphaned:
        if not quiet:
            print("No orphaned files to remove.")
        return 0

    if not quiet:
        print("\n\U0001f5d1\ufe0f  Removing %d orphaned files:" % len(orphaned))

    removed = 0
    for file_key in orphaned:
        if client.delete_object(config["bucket"], file_key):
            removed += 1
            state.remove_file(file_key)
            if not quiet:
                print("  \u2713 Removed %s" % file_key)
        else:
            if not quiet:
                print("  \u2717 Failed to remove %s" % file_key)

    state.save_state()

    if not quiet:
        print("\n\u2705 Removed %d orphaned files" % removed)

    return removed


def trigger_embedding(config, quiet=False):
    """Trigger embedding process on the bucket"""
    if not quiet:
        print("\n\U0001f9e0 Triggering embedding on bucket: %s" % config["bucket"])

    status, response = make_api_request("POST", "ai/embeddings", {
        "bucket_name": config["bucket"],
    })

    if status in [200, 201, 202]:
        if not quiet:
            print("\u2705 Embedding triggered successfully")
            if isinstance(response, dict) and "data" in response:
                task_id = response["data"].get("task_id")
                if task_id:
                    print("   Task ID: %s" % task_id)
                    print("   Check status with: ./sync.py --embed-status %s" % task_id)
        return True
    else:
        if not quiet:
            print("\u274c Failed to trigger embedding: %s" % status)
            print("   Response: %s" % response)
        return False


def check_embed_status(task_id, quiet=False):
    """Check embedding task status"""
    if not quiet:
        print("\n\U0001f50d Checking embedding task: %s" % task_id)

    status, response = make_api_request("GET", "ai/embeddings/%s" % task_id)

    if status == 200 and isinstance(response, dict):
        data = response.get("data", {})
        task_status = data.get("status", "unknown")
        progress = data.get("progress", 0)

        if not quiet:
            print("   Status: %s" % task_status)
            print("   Progress: %s%%" % progress)

            if task_status == "completed":
                print("\u2705 Embedding complete! You can now search.")
            elif task_status == "failed":
                error = data.get("error", "Unknown error")
                print("\u274c Error: %s" % error)
        return task_status
    else:
        if not quiet:
            print("\u274c Failed to check status: %s" % status)
        return "unknown"


def list_bucket_files(config, quiet=False):
    """List files in the bucket"""
    [REDACTED_SECRET])
    if not api_key:
        if not quiet:
            print("ERROR: No API key available")
        return []

    client = TelnyxS3Client(api_key, config["region"])
    files = client.list_objects(config["bucket"])

    if not quiet:
        print("\n\U0001f4c1 Files in %s (%d):" % (config["bucket"], len(files)))
        for f in sorted(files):
            print("  %s" % f)

    return files


def create_bucket(config, quiet=False):
    """Create the memory bucket"""
    [REDACTED_SECRET])
    if not api_key:
        if not quiet:
            print("ERROR: No API key available")
        return False

    client = TelnyxS3Client(api_key, config["region"])
    bucket = config["bucket"]

    if not quiet:
        print("Creating bucket: %s" % bucket)

    if client.create_bucket(bucket):
        if not quiet:
            print("\u2705 Bucket '%s' ready" % bucket)
        return True
    else:
        if not quiet:
            print("\u274c Failed to create bucket '%s'" % bucket)
        return False


def check_status(config, quiet=False):
    """Check bucket and embedding status"""
    [REDACTED_SECRET])
    if not api_key:
        if not quiet:
            print("ERROR: No API key available")
        return

    client = TelnyxS3Client(api_key, config["region"])

    if not quiet:
        print("\n\U0001f4ca Status for bucket: %s" % config["bucket"])

    if client.head_object(config["bucket"], ""):
        if not quiet:
            print("  \u2713 Bucket exists")
    else:
        if not quiet:
            print("  \u2717 Bucket not found")
        return

    files = client.list_objects(config["bucket"])
    if not quiet:
        print("  \U0001f4c1 Files indexed: %d" % len(files))

    if not quiet:
        print("  \U0001f9e0 Testing embeddings...")

    test_status, test_response = make_api_request("POST", "ai/embeddings/similarity-search", {
        "bucket_name": config["bucket"],
        "query": "test",
        "num_docs": 1,
    })

    if not quiet:
        if test_status == 200:
            print("  \u2705 Embeddings active and searchable")
        elif test_status == 404:
            print("  \u26a0\ufe0f Embeddings not enabled. Use --embed to trigger.")
        else:
            print("  \u2753 Embedding status unclear: %s" % test_status)


def watch_files(config, quiet=False):
    """Watch for file changes and sync in real-time"""
    if not quiet:
        print("\n\U0001f441\ufe0f Watching for changes in %s" % config["workspace"])
        print("   Press Ctrl+C to stop\n")

    state = SyncState()
    last_check = {}
    max_tokens = config.get("chunk_size", 800)

    try:
        while True:
            files = get_files_to_sync(config)
            changed = []

            for local_path, s3_key in files:
                try:
                    mtime = local_path.stat().st_mtime
                    if s3_key not in last_check or last_check[s3_key] != mtime:
                        if s3_key in last_check:
                            changed.append((local_path, s3_key))
                        last_check[s3_key] = mtime
                except OSError:
                    continue

            if changed:
                [REDACTED_SECRET])
                if [REDACTED_SECRET] = TelnyxS3Client(api_key, config["region"])
                    for local_path, s3_key in changed:
                        try:
                            chunks = chunk_file(local_path, s3_key, max_tokens)
                            if not chunks:
                                continue

                            # Delete old chunks
                            _delete_old_chunks(client, config["bucket"], state, s3_key, quiet)

                            chunk_keys = []
                            for ck, content, title in chunks:
                                if client.put_object(config["bucket"], ck, content):
                                    chunk_keys.append(ck)
                                    if not quiet:
                                        print("  \u2191 %s" % ck)
                                else:
                                    if not quiet:
                                        print("  \u2717 %s" % ck)

                            current_hash = calculate_file_hash(local_path)
                            if current_hash:
                                state.set_file_hash(s3_key, current_hash)
                            if len(chunks) > 1:
                                state.set_chunk_keys(s3_key, chunk_keys)
                            else:
                                state.remove_chunks(s3_key)
                        except Exception as e:
                            if not quiet:
                                print("  \u2717 %s (%s)" % (s3_key, e))

                    if changed:
                        state.save_state()

            time.sleep(2)
    except KeyboardInterrupt:
        if not quiet:
            print("\n\n\U0001f44b Stopped watching")


def main():
    parser = argparse.ArgumentParser(description="Telnyx RAG Memory Sync")
    parser.add_argument("--watch", "-w", action="store_true", help="Watch for changes")
    parser.add_argument("--list", "-l", action="store_true", help="List bucket files")
    parser.add_argument("--create-bucket", action="store_true", help="Create the bucket")
    parser.add_argument("--status", "-s", action="store_true", help="Check status")
    parser.add_argument("--prune", action="store_true", help="Remove orphaned files from bucket")
    parser.add_argument("--quiet", "-q", action="store_true", help="Only show errors")
    parser.add_argument("--embed", action="store_true", help="Trigger embedding after sync")
    parser.add_argument("--embed-status", metavar="TASK_ID", help="Check embedding task status")
    parser.add_argument(
        "--chunk-size", type=int, default=None,
        help="Override chunk size in tokens (default: from config, approx 800)"
    )

    args = parser.parse_args()
    config = load_config()

    if not args.quiet:
        print("\n\U0001f9e0 Telnyx RAG Memory Sync\n" + "=" * 40)

    try:
        if args.create_bucket:
            create_bucket(config, args.quiet)
        elif args.list:
            list_bucket_files(config, args.quiet)
        elif args.status:
            check_status(config, args.quiet)
        elif args.prune:
            prune_orphaned_files(config, args.quiet)
        elif args.embed and not args.watch:
            trigger_embedding(config, args.quiet)
        elif args.embed_status:
            check_embed_status(args.embed_status, args.quiet)
        elif args.watch:
            sync_files(config, args.quiet, chunk_size_override=args.chunk_size)
            watch_files(config, args.quiet)
        else:
            success, failed = sync_files(
                config, args.quiet, chunk_size_override=args.chunk_size
            )
            if args.embed and success > 0:
                trigger_embedding(config, args.quiet)
    except KeyboardInterrupt:
        if not args.quiet:
            print("\n\nOperation cancelled by user")
        sys.exit(1)
    except Exception as e:
        print("ERROR: %s" % e, file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)