Skill: simondelorean/gradient-knowledge-base (version 0.1.4)

--- name: gradient-knowledge-base description: > Community skill (unofficial) for DigitalOcean Gradient Knowledge Bases. Build RAG pipelines: store documents in DO Spaces, configure data sources, manage indexing, and run semantic or hybrid search queries. files: ["scripts/*"] homepage: https://github.com/Rogue-Iteration/TheBigClaw metadata: clawdbot: emoji: "📚" primaryEnv: DO_API_TOKEN requires: env: - DO_API_TOKEN - DO_SPACES_ACCESS_KEY - DO_SPACES_SECRET_KEY - GRADIENT_API_KEY bins: - python3 pip: - requests>=2.31.0 - boto3>=1.34.0 author: Rogue Iteration version: "0.1.4" tags: ["digitalocean", "gradient-ai", "knowledge-base", "rag", "semantic-search", "do-spaces"] --- # 🦞 Gradient AI — Knowledge Bases & RAG > ⚠️ **This is an unofficial community skill**, not maintained by DigitalOcean. Use at your own risk. > *"A lobster never forgets. Neither should your agent." — the KB lobster* Build a [Retrieval-Augmented Generation](https://docs.digitalocean.com/products/gradient-ai-platform/details/features/#retrieval-augmented-generation-rag) pipeline using DigitalOcean's Gradient Knowledge Bases. Store your documents in DO Spaces, index them into a managed Knowledge Base (backed by OpenSearch), and query them with semantic or hybrid search. ## Architecture ``` Your Agent DigitalOcean ┌─────────────┐ upload ┌──────────────┐ │ Documents │ ──────────▶ │ DO Spaces │ └─────────────┘ │ (S3-compat) │ └──────┬───────┘ │ auto-index ┌──────▼───────┐ │ Knowledge │ │ Base (KBaaS) │ │ ┌──────────┐ │ │ │OpenSearch│ │ │ └──────────┘ │ └──────┬───────┘ │ retrieve ┌─────────────┐ answer ┌──────▼───────┐ │ Your Agent │ ◀────────── │ RAG Results │ │ + LLM │ │ + Citations │ └─────────────┘ └──────────────┘ ``` 📖 *[Knowledge Base docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/)* ## API Endpoints This skill connects to three official DigitalOcean service endpoints: | Hostname | Purpose | Docs | |----------|---------|------| | `api.digitalocean.com` | KB management (create, list, delete, data sources) | [DO API Reference](https://docs.digitalocean.com/reference/api/) | | `kbaas.do-ai.run` | KB retrieval — semantic/hybrid search queries | [KB Retrieval docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/) | | `inference.do-ai.run` | LLM chat completions for RAG synthesis | [Inference docs](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/) | | `<region>.digitaloceanspaces.com` | S3-compatible object storage | [Spaces docs](https://docs.digitalocean.com/products/spaces/) | All endpoints are owned and operated by DigitalOcean. The `*.do-ai.run` hostnames are the Gradient AI Platform's service domains. ## Authentication This skill uses **two different credentials** — think of it as a two-claw approach: | Credential | Used For | Env Var | |------------|----------|---------| | DO API Token | KB management, indexing, queries | `DO_API_TOKEN` | | Gradient API Key | LLM inference for RAG synthesis | `GRADIENT_API_KEY` | | Spaces Keys | S3-compatible uploads | `DO_SPACES_ACCESS_KEY` + `DO_SPACES_SECRET_KEY` | > **Credential scoping:** Use minimally-scoped tokens. Create a dedicated [Model Access Key](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/manage-access-keys/) for `GRADIENT_API_KEY`. For `DO_API_TOKEN`, use a [scoped API token](https://docs.digitalocean.com/reference/api/create-personal-access-token/) with only Knowledge Base and Spaces permissions. Avoid using your account-root token. Optional but recommended: ```bash export GRADIENT_KB_UUID="your-kb-uuid" # Default KB for queries export DO_SPACES_BUCKET="your-bucket" # Default bucket for uploads export DO_SPACES_ENDPOINT="https://nyc3.digitaloceanspaces.com" ``` --- ## Tools ### 📦 Store Documents in Spaces Upload files to DO Spaces for Knowledge Base indexing. This is the storage layer — documents land here before being indexed. ```bash # Upload a file python3 gradient_spaces.py --upload /path/to/report.md --bucket my-kb-data # Upload with a key prefix (folder structure) python3 gradient_spaces.py --upload report.md --bucket my-kb-data --prefix "research/2026-02-15/" # List files in a bucket python3 gradient_spaces.py --list --bucket my-kb-data # List files with a prefix filter python3 gradient_spaces.py --list --bucket my-kb-data --prefix "research/" # Delete a file python3 gradient_spaces.py --delete "research/old_report.md" --bucket my-kb-data ``` 📖 *[DO Spaces docs](https://docs.digitalocean.com/products/spaces/)* --- ### 🏗️ Create and Manage Knowledge Bases Full CRUD for Knowledge Bases. Create them programmatically instead of clicking through the console like a land-dweller. ```bash # List all Knowledge Bases python3 gradient_kb_manage.py --list # Create a new KB python3 gradient_kb_manage.py --create --name "My Research KB" --region nyc3 # Show details for a specific KB python3 gradient_kb_manage.py --show --kb-uuid "your-kb-uuid" # Delete a KB (⚠️ permanent!) python3 gradient_kb_manage.py --delete --kb-uuid "your-kb-uuid" ``` 📖 *[Create KBs via API](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/)* --- ### 📁 Manage Data Sources Connect your Spaces bucket (or web URLs) to a Knowledge Base. This is what tells the KB "index these documents." ```bash # Add a DO Spaces data source python3 gradient_kb_manage.py --add-source \ --kb-uuid "your-kb-uuid" \ --bucket my-kb-data \ --prefix "research/" # List data sources for a KB python3 gradient_kb_manage.py --list-sources --kb-uuid "your-kb-uuid" # Trigger re-indexing (auto-detects the data source) python3 gradient_kb_manage.py --reindex --kb-uuid "your-kb-uuid" # Trigger re-indexing for a specific source python3 gradient_kb_manage.py --reindex --kb-uuid "your-kb-uuid" --source-uuid "ds-uuid" ``` > **🦞 Pro tip: Auto-indexing.** If your KB has auto-indexing enabled, you can skip manual re-index triggers. The KB will detect changes in your Spaces bucket automatically. Configure it in the [DigitalOcean Console](https://cloud.digitalocean.com) → Knowledge Base → Settings. --- ### 🔍 Query the Knowledge Base Search your indexed documents with semantic or hybrid queries. This is where the magic happens — your documents become answers. ```bash # Basic query python3 gradient_kb_query.py --query "What happened with the Q4 earnings?" # Control number of results python3 gradient_kb_query.py --query "Revenue trends" --num-results 20 # Tune hybrid search balance (see below) python3 gradient_kb_query.py --query "$CAKE price movement" --alpha 0.5 # JSON output (for piping to other tools) python3 gradient_kb_query.py --query "SEC filings summary" --json ``` **Direct API call:** ```bash curl -s https://kbaas.do-ai.run/v1/{kb-uuid}/retrieve \ -H "Authorization: Bearer $DO_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "query": "What happened with Q4 earnings?", "num_results": 10, "alpha": 0.5 }' ``` 📖 *[KB retrieval API](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/#query-a-knowledge-base)* --- ### 🎛️ The `alpha` Parameter — Hybrid Search Tuning This is the secret sauce. The `alpha` parameter controls the balance between **lexical** (keyword) and **semantic** (meaning) search: | Alpha | Behavior | Best For | |-------|----------|----------| | `0.0` | Pure lexical (keyword matching) | Exact terms: ticker symbols, filing numbers, dates | | `0.5` | Balanced hybrid | General research queries | | `1.0` | Pure semantic (meaning-based) | Open-ended: "what happened with...", "summarize..." | > **🦞 Rule of claw:** Start at `0.5`. Go lower when searching for specific things (`$CAKE`, `10-K`, `2026-02-15`). Go higher when exploring ideas ("What's the market sentiment?"). --- ### 🧠 RAG-Enhanced Queries The full pipeline: query the KB → build a context prompt → call an LLM to synthesize. One command, complete answers with citations. ```bash python3 gradient_kb_query.py \ --query "Summarize all research on $CAKE" \ --rag \ --model "openai-gpt-oss-120b" ``` This automatically: 1. 🔍 Queries the Knowledge Base for relevant documents 2. 📝 Builds a prompt with the retrieved context 3. 🤖 Calls the LLM to synthesize an answer > **Note:** RAG queries call the [Gradient Inference API](https://docs.digitalocean.com/products/gradient-ai-platform/how-to/use-serverless-inference/) under the hood, so you'll need `GRADIENT_API_KEY` set. If you have the `gradient-inference` skill loaded too, you're all set. --- ## Advanced Configuration ### Embedding Models & Chunking When creating a Knowledge Base, you can choose how documents are split into searchable chunks: | Strategy | How It Works | Best For | |----------|-------------|----------| | **Section-based** | Splits on document structure (headings, paragraphs) | Structured reports | | **Semantic** | Splits on meaning boundaries | Narrative content | | **Hierarchical** | Preserves document hierarchy in chunks | Technical docs | | **Fixed-length** | Equal-sized chunks | Uniform data | Configure these in the [DigitalOcean Console](https://cloud.digitalocean.com) when creating the KB, or via the API's `embedding_model` and chunking parameters. 📖 *[KB configuration options](https://docs.digitalocean.com/products/gradient-ai-platform/details/features/#retrieval-augmented-generation-rag)* --- ## CLI Reference All scripts accept `--json` for machine-readable output. ``` gradient_spaces.py --upload FILE | --list | --delete KEY [--bucket NAME] [--prefix PATH] [--key KEY] [--json] gradient_kb_manage.py --list | --create | --show | --delete | --list-sources | --add-source | --reindex [--kb-uuid UUID] [--source-uuid UUID] [--name NAME] [--region REGION] [--bucket NAME] [--prefix PATH] [--json] gradient_kb_query.py --query TEXT [--kb-uuid UUID] [--num-results N] [--alpha F] [--rag] [--model ID] [--json] ``` ## Environment Variables | Variable | Required | Description | |----------|----------|-------------| | `DO_API_TOKEN` | ✅ | DO API token (scopes: GenAI + Spaces) | | `DO_SPACES_ACCESS_KEY` | ✅ | Spaces access key | | `DO_SPACES_SECRET_KEY` | ✅ | Spaces secret key | | `DO_SPACES_ENDPOINT` | Optional | Spaces endpoint (default: `https://nyc3.digitaloceanspaces.com`) | | `DO_SPACES_BUCKET` | Optional | Default bucket name | | `GRADIENT_KB_UUID` | Optional | Default KB UUID (saves typing `--kb-uuid` every time) | | `GRADIENT_API_KEY` | For RAG | Needed when using `--rag` for LLM synthesis | ## External Endpoints | Endpoint | Purpose | |----------|---------| | `https://kbaas.do-ai.run/v1/{uuid}/retrieve` | KB retrieval API | | `https://api.digitalocean.com/v2/gen-ai/knowledge_bases/` | KB management API | | `https://{region}.digitaloceanspaces.com` | DO Spaces (S3-compatible) | ## Security & Privacy - Your `DO_API_TOKEN` is sent as a Bearer token to `api.digitalocean.com` and `kbaas.do-ai.run` - Spaces credentials are used for S3-compatible uploads to `{region}.digitaloceanspaces.com` - Documents you upload become **private** in your Spaces bucket by default - KB queries are scoped to your account — no cross-tenant access - No credentials or data are sent to any third-party endpoints ## Trust Statement > By using this skill, documents and queries are sent to DigitalOcean's Knowledge Base > and Spaces APIs. Only install if you trust DigitalOcean with the documents you index. ## Important Notes - Documents uploaded to Spaces are **private by default** - Re-indexing is **best-effort** — if the API call fails, auto-indexing kicks in on its own schedule - The retrieval API returns document **chunks**, not full documents - Deleting a KB is **permanent** — the indexed data is gone. The source files in Spaces are not affected.

path: requirements.txt
content: # Python dependencies for the gradient-knowledge-base skill
# Install in a virtualenv: pip install -r requirements.txt
requests>=2.31.0
boto3>=1.34.0


path: scripts/gradient_kb_manage.py
content: #!/usr/bin/env python3
"""
🦞 Gradient AI — Knowledge Base Management

Create, list, and manage DigitalOcean Gradient Knowledge Bases
and their data sources. Full CRUD via the DO API.

Usage:
    python3 gradient_kb_manage.py --list
    python3 gradient_kb_manage.py --create --name "My Research KB"
    python3 gradient_kb_manage.py --show --kb-uuid <uuid>
    python3 gradient_kb_manage.py --add-source --kb-uuid <uuid> --type spaces --bucket my-data
    python3 gradient_kb_manage.py --reindex --kb-uuid <uuid>

Docs: https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/
API:  https://docs.digitalocean.com/reference/api/digitalocean-api/
"""

import argparse
import json
import os
import sys
from typing import Optional

import requests

DO_API_BASE = "https://api.digitalocean.com"
KB_API_PATH = "/v2/gen-ai/knowledge_bases"


def _headers(api_token: str) -> dict:
    """Build auth headers for the DO API."""
    return {
        "Authorization": f"Bearer {api_token}",
        "Content-Type": "application/json",
    }


def _resolve_token(api_token: Optional[str] = None) -> str:
    """Resolve the API token from arg or env."""
    return api_token or os.environ.get("DO_API_TOKEN", "")


def list_knowledge_bases(api_token: Optional[str] = None) -> dict:
    """List all Knowledge Bases in your account.

    Args:
        api_token: DO API token. Falls back to DO_API_TOKEN.

    Returns:
        dict with 'success', 'knowledge_bases' (list), and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "knowledge_bases": [], "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}"
        resp = requests.get(url, headers=_headers(token), timeout=15)
        resp.raise_for_status()

        data = resp.json()
        kbs = data.get("knowledge_bases", [])

        return {
            "success": True,
            "knowledge_bases": kbs,
            "message": f"Found {len(kbs)} Knowledge Base(s).",
        }
    except requests.RequestException as e:
        return {"success": False, "knowledge_bases": [], "message": f"API request failed: {str(e)}"}


def create_knowledge_base(
    name: str,
    region: str = "nyc3",
    project_id: Optional[str] = None,
    embedding_model: Optional[str] = None,
    api_token: Optional[str] = None,
) -> dict:
    """Create a new Knowledge Base.

    Args:
        name: Human-readable name for the KB.
        region: DO region (e.g., 'nyc3').
        project_id: DO project UUID to associate with.
        embedding_model: Embedding model ID (uses default if not specified).
        api_token: DO API token.

    Returns:
        dict with 'success', 'knowledge_base' (created KB data), and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "knowledge_base": {}, "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}"
        payload = {
            "name": name,
            "region": region,
        }

        if project_id:
            payload["project_id"] = project_id
        if embedding_model:
            payload["embedding_model"] = embedding_model

        resp = requests.post(url, headers=_headers(token), json=payload, timeout=30)
        resp.raise_for_status()

        data = resp.json()
        kb = data.get("knowledge_base", data)

        return {
            "success": True,
            "knowledge_base": kb,
            "message": f"Created Knowledge Base '{name}' (UUID: {kb.get('uuid', 'unknown')}).",
        }
    except requests.RequestException as e:
        return {"success": False, "knowledge_base": {}, "message": f"Create failed: {str(e)}"}


def get_knowledge_base(
    kb_uuid: str,
    api_token: Optional[str] = None,
) -> dict:
    """Get details for a specific Knowledge Base.

    Args:
        kb_uuid: Knowledge Base UUID.
        api_token: DO API token.

    Returns:
        dict with 'success', 'knowledge_base', and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "knowledge_base": {}, "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}/{kb_uuid}"
        resp = requests.get(url, headers=_headers(token), timeout=15)
        resp.raise_for_status()

        data = resp.json()
        kb = data.get("knowledge_base", data)

        return {
            "success": True,
            "knowledge_base": kb,
            "message": f"Knowledge Base: {kb.get('name', kb_uuid)}",
        }
    except requests.RequestException as e:
        return {"success": False, "knowledge_base": {}, "message": f"Get failed: {str(e)}"}


def delete_knowledge_base(
    kb_uuid: str,
    api_token: Optional[str] = None,
) -> dict:
    """Delete a Knowledge Base.

    ⚠️ This permanently deletes the KB and all its indexed data.

    Args:
        kb_uuid: Knowledge Base UUID.
        api_token: DO API token.

    Returns:
        dict with 'success' and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}/{kb_uuid}"
        resp = requests.delete(url, headers=_headers(token), timeout=15)
        resp.raise_for_status()

        return {
            "success": True,
            "message": f"Deleted Knowledge Base {kb_uuid}.",
        }
    except requests.RequestException as e:
        return {"success": False, "message": f"Delete failed: {str(e)}"}


def list_data_sources(
    kb_uuid: str,
    api_token: Optional[str] = None,
) -> dict:
    """List data sources for a Knowledge Base.

    Args:
        kb_uuid: Knowledge Base UUID.
        api_token: DO API token.

    Returns:
        dict with 'success', 'data_sources' (list), and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "data_sources": [], "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}/{kb_uuid}/data_sources"
        resp = requests.get(url, headers=_headers(token), timeout=15)
        resp.raise_for_status()

        data = resp.json()
        sources = data.get("knowledge_base_data_sources", data.get("data_sources", []))

        return {
            "success": True,
            "data_sources": sources,
            "message": f"Found {len(sources)} data source(s).",
        }
    except requests.RequestException as e:
        return {"success": False, "data_sources": [], "message": f"List sources failed: {str(e)}"}


def add_spaces_source(
    kb_uuid: str,
    bucket: str,
    prefix: str = "",
    region: str = "nyc3",
    api_token: Optional[str] = None,
) -> dict:
    """Add a DO Spaces bucket as a data source for a Knowledge Base.

    Args:
        kb_uuid: Knowledge Base UUID.
        bucket: Spaces bucket name.
        prefix: Key prefix to scope indexing to a specific folder.
        region: Spaces region.
        api_token: DO API token.

    Returns:
        dict with 'success', 'data_source', and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "data_source": {}, "message": "No DO_API_TOKEN configured."}

    try:
        url = f"{DO_API_BASE}{KB_API_PATH}/{kb_uuid}/data_sources"
        payload = {
            "type": "spaces",
            "spaces": {
                "bucket": bucket,
                "region": region,
            },
        }

        if prefix:
            payload["spaces"]["prefix"] = prefix

        resp = requests.post(url, headers=_headers(token), json=payload, timeout=15)
        resp.raise_for_status()

        data = resp.json()
        source = data.get("knowledge_base_data_source", data)

        return {
            "success": True,
            "data_source": source,
            "message": f"Added Spaces source ({bucket}/{prefix}) to KB {kb_uuid}.",
        }
    except requests.RequestException as e:
        return {"success": False, "data_source": {}, "message": f"Add source failed: {str(e)}"}


def trigger_reindex(
    kb_uuid: str,
    source_uuid: Optional[str] = None,
    api_token: Optional[str] = None,
) -> dict:
    """Trigger a re-indexing job for a Knowledge Base data source.

    If source_uuid is not provided, fetches the first data source and
    triggers re-indexing on it.

    Args:
        kb_uuid: Knowledge Base UUID.
        source_uuid: Data source UUID (optional — auto-detected if omitted).
        api_token: DO API token.

    Returns:
        dict with 'success' and 'message'.
    """
    [REDACTED_SECRET])
    if not [REDACTED_SECRET] {"success": False, "message": "No DO_API_TOKEN configured."}

    try:
        # Auto-detect source UUID if not provided
        if not source_uuid:
            sources_result = list_data_sources(kb_uuid, api_token=token)
            if not sources_result["success"]:
                return sources_result
            sources = sources_result["data_sources"]
            if not sources:
                return {"success": False, "message": "No data sources configured for this Knowledge Base."}
            source_uuid = sources[0].get("uuid", "")
            if not source_uuid:
                return {"success": False, "message": "Data source has no UUID."}

        url = f"{DO_API_BASE}{KB_API_PATH}/{kb_uuid}/data_sources/{source_uuid}/indexing_jobs"
        resp = requests.post(url, headers=_headers(token), json={}, timeout=15)
        resp.raise_for_status()

        return {
            "success": True,
            "message": f"Re-indexing triggered for KB {kb_uuid}, data source {source_uuid}.",
        }
    except requests.RequestException:
        # Best-effort — auto-indexing may already be configured
        return {
            "success": True,
            "message": "Re-index request sent. If auto-indexing is enabled, the KB will update on its own schedule.",
        }


# ─── CLI Interface ────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(
        description="🦞 Manage Gradient Knowledge Bases"
    )
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--list", action="store_true", help="List all Knowledge Bases")
    group.add_argument("--create", action="store_true", help="Create a new Knowledge Base")
    group.add_argument("--show", action="store_true", help="Show KB details")
    group.add_argument("--delete", action="store_true", help="Delete a Knowledge Base")
    group.add_argument("--list-sources", action="store_true", help="List data sources for a KB")
    group.add_argument("--add-source", action="store_true", help="Add a data source to a KB")
    group.add_argument("--reindex", action="store_true", help="Trigger re-indexing")

    parser.add_argument("--kb-uuid", default=None, help="Knowledge Base UUID")
    parser.add_argument("--source-uuid", default=None, help="Data source UUID")
    parser.add_argument("--name", default=None, help="KB name (for create)")
    parser.add_argument("--region", default="nyc3", help="Region (default: nyc3)")
    parser.add_argument("--project-id", default=None, help="DO project UUID")
    parser.add_argument("--type", dest="source_type", default="spaces", help="Data source type")
    parser.add_argument("--bucket", default=None, help="Spaces bucket name")
    parser.add_argument("--prefix", default="", help="Spaces key prefix")
    parser.add_argument("--json", action="store_true", help="Output as JSON")

    args = parser.parse_args()

    result = None

    if args.list:
        result = list_knowledge_bases()
    elif args.create:
        if not args.name:
            print("Error: --name required for --create", file=sys.stderr)
            sys.exit(1)
        result = create_knowledge_base(name=args.name, region=args.region, project_id=args.project_id)
    elif args.show:
        if not args.kb_uuid:
            print("Error: --kb-uuid required for --show", file=sys.stderr)
            sys.exit(1)
        result = get_knowledge_base(args.kb_uuid)
    elif args.delete:
        if not args.kb_uuid:
            print("Error: --kb-uuid required for --delete", file=sys.stderr)
            sys.exit(1)
        result = delete_knowledge_base(args.kb_uuid)
    elif args.list_sources:
        if not args.kb_uuid:
            print("Error: --kb-uuid required for --list-sources", file=sys.stderr)
            sys.exit(1)
        result = list_data_sources(args.kb_uuid)
    elif args.add_source:
        if not args.kb_uuid or not args.bucket:
            print("Error: --kb-uuid and --bucket required for --add-source", file=sys.stderr)
            sys.exit(1)
        result = add_spaces_source(args.kb_uuid, args.bucket, prefix=args.prefix, region=args.region)
    elif args.reindex:
        if not args.kb_uuid:
            print("Error: --kb-uuid required for --reindex", file=sys.stderr)
            sys.exit(1)
        result = trigger_reindex(args.kb_uuid, source_uuid=args.source_uuid)

    if result:
        if args.json:
            print(json.dumps(result, indent=2, default=str))
        else:
            if result["success"]:
                print(f"✅ {result['message']}")
                # Print KBs if listing
                if "knowledge_bases" in result:
                    for kb in result["knowledge_bases"]:
                        name = kb.get("name", "unnamed")
                        uuid = kb.get("uuid", "?")
                        print(f"  📦 {name} ({uuid})")
                # Print data sources if listing
                if "data_sources" in result:
                    for ds in result["data_sources"]:
                        ds_type = ds.get("type", "unknown")
                        uuid = ds.get("uuid", "?")
                        print(f"  📁 {ds_type} ({uuid})")
            else:
                print(f"❌ {result['message']}", file=sys.stderr)
                sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/gradient_kb_query.py
content: #!/usr/bin/env python3
"""
🦞 Gradient AI — Knowledge Base Query

Query a DigitalOcean Gradient Knowledge Base with semantic or hybrid search.
Optionally combine with an LLM call for full RAG (Retrieval-Augmented Generation).

Usage:
    python3 gradient_kb_query.py --query "What happened with Q4 earnings?"
    python3 gradient_kb_query.py --query "$CAKE analysis" --alpha 0.5
    python3 gradient_kb_query.py --query "Summarize CAKE" --rag --model openai-gpt-oss-120b

Docs: https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/
"""

import argparse
import json
import os
import sys
from typing import Optional

import requests

KB_RETRIEVE_BASE_URL = "https://kbaas.do-ai.run/v1"
INFERENCE_URL = "https://inference.do-ai.run/v1/chat/completions"


def query_kb(
    query: str,
    kb_uuid: Optional[str] = None,
    api_token: Optional[str] = None,
    num_results: int = 10,
    alpha: Optional[float] = None,
) -> dict:
    """Query a Gradient Knowledge Base.

    Uses the KB retrieval endpoint for semantic or hybrid search.

    Args:
        query: The search query.
        kb_uuid: Knowledge Base UUID. Falls back to GRADIENT_KB_UUID.
        api_token: DO API token. Falls back to DO_API_TOKEN.
        num_results: Number of results to return (1–100).
        alpha: Hybrid search tuning (0.0=lexical, 0.5=balanced, 1.0=semantic).
               None uses the API default.

    Returns:
        dict with 'success', 'results' (list), 'query', and 'message'.

    See: https://docs.digitalocean.com/products/gradient-ai-platform/how-to/create-manage-knowledge-bases/
    """
    kb_uuid = kb_uuid or os.environ.get("GRADIENT_KB_UUID", "")
    api_token = api_token or os.environ.get("DO_API_TOKEN", "")

    if not kb_uuid:
        return {"success": False, "results": [], "query": query, "message": "No GRADIENT_KB_UUID configured."}
    if not api_token:
        return {"success": False, "results": [], "query": query, "message": "No DO_API_TOKEN configured."}

    try:
        headers = {
            "Authorization": f"Bearer {api_token}",
            "Content-Type": "application/json",
        }

        url = f"{KB_RETRIEVE_BASE_URL}/{kb_uuid}/retrieve"
        payload = {"query": query, "num_results": num_results}

        if alpha is not None:
            payload["alpha"] = alpha

        resp = requests.post(url, headers=headers, json=payload, timeout=15)
        resp.raise_for_status()

        data = resp.json()
        results = data.get("results", [])

        return {
            "success": True,
            "results": results,
            "query": query,
            "message": f"Found {len(results)} relevant documents.",
        }
    except requests.RequestException as e:
        return {"success": False, "results": [], "query": query, "message": f"KB query failed: {str(e)}"}


def build_rag_messages(query: str, kb_results: list[dict]) -> list[dict]:
    """Build structured messages for RAG, separating system context from user input.

    Returns a list of chat messages with proper role separation to prevent
    prompt injection — user input stays in the 'user' role, instructions
    and KB context stay in the 'system' role.

    Args:
        query: The user's original question.
        kb_results: Results from the Knowledge Base query.

    Returns:
        List of message dicts with 'role' and 'content' keys.
    """
    if not kb_results:
        context = "No relevant documents found in the knowledge base yet. It may still be building up."
    else:
        context_parts = []
        for i, result in enumerate(kb_results, 1):
            content = result.get("content", result.get("text", ""))
            source = result.get("metadata", {}).get("source", "unknown")
            score = result.get("score", 0)
            context_parts.append(f"### Source {i} (relevance: {score:.2f}, source: {source})\n{content}")
        context = "\n\n".join(context_parts)

    system_msg = f"""You are a helpful assistant with access to a knowledge base.
Answer the user's question using ONLY the retrieved context below.
Be specific, cite source numbers when available, and note when information might be incomplete.
If the KB doesn't have enough data, say so honestly. Be concise but thorough.

## Retrieved Context (from Knowledge Base):
{context}"""

    return [
        {"role": "system", "content": system_msg},
        {"role": "user", "content": query},
    ]


def query_with_rag(
    query: str,
    kb_uuid: Optional[str] = None,
    model: str = "openai-gpt-oss-120b",
    [REDACTED_SECRET] = None,
    api_token: Optional[str] = None,
    num_results: int = 10,
    alpha: Optional[float] = None,
) -> dict:
    """Run a full RAG pipeline: KB search → prompt construction → LLM synthesis.

    Args:
        query: The user's question.
        kb_uuid: Knowledge Base UUID. Falls back to GRADIENT_KB_UUID.
        model: LLM model for synthesis.
        [REDACTED_SECRET] API key (for inference). Falls back to GRADIENT_API_KEY.
        api_token: DO API token (for KB query). Falls back to DO_API_TOKEN.
        num_results: Number of KB results to retrieve.
        alpha: Hybrid search tuning parameter.

    Returns:
        dict with 'success', 'answer', 'sources_count', 'kb_success', and 'message'.
    """
    [REDACTED_SECRET] or os.environ.get("GRADIENT_API_KEY", "")

    if not [REDACTED_SECRET] {
            "success": False,
            "answer": "",
            "sources_count": 0,
            "message": "No GRADIENT_API_KEY configured for LLM synthesis.",
        }

    # Step 1: Query the Knowledge Base
    kb_result = query_kb(query, kb_uuid=kb_uuid, api_token=api_token,
                         num_results=num_results, alpha=alpha)
    kb_results = kb_result.get("results", [])

    # Step 2: Build structured messages (system context + user query separated)
    messages = build_rag_messages(query, kb_results)

    # Step 3: Call LLM for synthesis
    try:
        headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        }

        payload = {
            "model": model,
            "messages": messages,
            "temperature": 0.4,
            "max_tokens": 1500,
        }

        resp = requests.post(INFERENCE_URL, headers=headers, json=payload, timeout=30)
        resp.raise_for_status()

        data = resp.json()
        answer = data["choices"][0]["message"]["content"]

        return {
            "success": True,
            "answer": answer,
            "sources_count": len(kb_results),
            "kb_success": kb_result["success"],
            "message": f"Answered using {len(kb_results)} sources from the Knowledge Base.",
        }
    except (requests.RequestException, KeyError, IndexError) as e:
        return {
            "success": False,
            "answer": "",
            "sources_count": len(kb_results),
            "message": f"LLM synthesis failed: {str(e)}",
        }


# ─── CLI Interface ────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(
        description="🦞 Query a Gradient Knowledge Base"
    )
    parser.add_argument("--query", required=True, help="Search query")
    parser.add_argument("--kb-uuid", default=None, help="Knowledge Base UUID")
    parser.add_argument("--num-results", type=int, default=10, help="Number of results (default: 10)")
    parser.add_argument("--alpha", type=float, default=None,
                        help="Hybrid search tuning: 0.0=lexical, 0.5=balanced, 1.0=semantic")
    parser.add_argument("--rag", action="store_true", help="Run full RAG pipeline (KB + LLM)")
    parser.add_argument("--model", default="openai-gpt-oss-120b", help="LLM model for RAG synthesis")
    parser.add_argument("--json", action="store_true", help="Output as JSON")

    args = parser.parse_args()

    if args.rag:
        result = query_with_rag(
            query=args.query,
            kb_uuid=args.kb_uuid,
            model=args.model,
            num_results=args.num_results,
            alpha=args.alpha,
        )
        if args.json:
            print(json.dumps(result, indent=2))
        elif result["success"]:
            print(result["answer"])
            print(f"\n📚 Used {result['sources_count']} sources.", file=sys.stderr)
        else:
            print(f"Error: {result['message']}", file=sys.stderr)
            sys.exit(1)
    else:
        result = query_kb(
            query=args.query,
            kb_uuid=args.kb_uuid,
            num_results=args.num_results,
            alpha=args.alpha,
        )
        if args.json:
            print(json.dumps(result, indent=2, default=str))
        elif result["success"]:
            for i, r in enumerate(result["results"], 1):
                content = r.get("content", r.get("text", ""))
                score = r.get("score", 0)
                print(f"[{i}] (score: {score:.2f}) {content[:200]}...")
            print(f"\n🔍 {result['message']}", file=sys.stderr)
        else:
            print(f"Error: {result['message']}", file=sys.stderr)
            sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/gradient_spaces.py
content: #!/usr/bin/env python3
"""
🦞 Gradient AI — DO Spaces Operations

Upload, list, and delete files in DigitalOcean Spaces (S3-compatible).
This is the storage layer for the Knowledge Base pipeline — documents
land here before being indexed.

Usage:
    python3 gradient_spaces.py --upload /path/to/report.md --bucket my-data
    python3 gradient_spaces.py --list --bucket my-data --prefix "research/"
    python3 gradient_spaces.py --delete "research/old.md" --bucket my-data

Docs: https://docs.digitalocean.com/products/spaces/
"""

import argparse
import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional

import boto3
import requests
from botocore.config import Config


def get_spaces_client(
    access_key: Optional[str] = None,
    secret_key: Optional[str] = None,
    endpoint: Optional[str] = None,
):
    """Create an S3-compatible client for DO Spaces.

    Falls back to environment variables if args aren't provided.

    Args:
        access_key: Spaces access key. Falls back to DO_SPACES_ACCESS_KEY.
        secret_key: Spaces secret key. Falls back to DO_SPACES_SECRET_KEY.
        endpoint: Spaces endpoint URL. Falls back to DO_SPACES_ENDPOINT.

    Returns:
        boto3 S3 client configured for DO Spaces.
    """
    access_key = access_key or os.environ.get("DO_SPACES_ACCESS_KEY", "")
    secret_key = secret_key or os.environ.get("DO_SPACES_SECRET_KEY", "")
    endpoint = endpoint or os.environ.get("DO_SPACES_ENDPOINT", "https://nyc3.digitaloceanspaces.com")

    return boto3.client(
        "s3",
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        config=Config(signature_version="s3v4"),
    )


def build_key(prefix: str, filename: str) -> str:
    """Build the S3 key (path) for a file.

    Args:
        prefix: Key prefix (folder path), e.g., 'research/2026-02-15/'
        filename: File name to append.

    Returns:
        The full S3 key string.
    """
    prefix = prefix.rstrip("/")
    if prefix:
        return f"{prefix}/{filename}"
    return filename


def upload_file(
    content: str,
    key: str,
    bucket: Optional[str] = None,
    client=None,
    content_type: str = "text/markdown",
) -> dict:
    """Upload content to DO Spaces.

    Args:
        content: The text content to upload.
        key: S3 key (full path in the bucket).
        bucket: Bucket name. Falls back to DO_SPACES_BUCKET.
        client: Pre-configured S3 client (optional).
        content_type: MIME type of the content.

    Returns:
        dict with 'success', 'key', 'bucket', and 'message'.
    """
    bucket = bucket or os.environ.get("DO_SPACES_BUCKET", "")

    if not bucket:
        return {"success": False, "key": key, "bucket": "", "message": "No bucket specified."}

    try:
        if client is None:
            client = get_spaces_client()

        client.put_object(
            Bucket=bucket,
            Key=key,
            Body=content.encode("utf-8"),
            ContentType=content_type,
            ACL="private",
        )

        return {
            "success": True,
            "key": key,
            "bucket": bucket,
            "message": f"Uploaded {key} to {bucket}",
        }
    except Exception as e:
        return {
            "success": False,
            "key": key,
            "bucket": bucket,
            "message": f"Upload failed: {str(e)}",
        }


def list_files(
    bucket: Optional[str] = None,
    prefix: str = "",
    client=None,
) -> dict:
    """List files in a DO Spaces bucket.

    Args:
        bucket: Bucket name. Falls back to DO_SPACES_BUCKET.
        prefix: Only list files with this key prefix.
        client: Pre-configured S3 client (optional).

    Returns:
        dict with 'success', 'files' (list of dicts), and 'message'.
    """
    bucket = bucket or os.environ.get("DO_SPACES_BUCKET", "")

    if not bucket:
        return {"success": False, "files": [], "message": "No bucket specified."}

    try:
        if client is None:
            client = get_spaces_client()

        kwargs = {"Bucket": bucket}
        if prefix:
            kwargs["Prefix"] = prefix

        resp = client.list_objects_v2(**kwargs)
        contents = resp.get("Contents", [])

        files = [
            {
                "key": obj["Key"],
                "size": obj["Size"],
                "last_modified": obj["LastModified"].isoformat() if hasattr(obj["LastModified"], "isoformat") else str(obj["LastModified"]),
            }
            for obj in contents
        ]

        return {
            "success": True,
            "files": files,
            "message": f"Found {len(files)} file(s) in {bucket}/{prefix}",
        }
    except Exception as e:
        return {"success": False, "files": [], "message": f"List failed: {str(e)}"}


def delete_file(
    key: str,
    bucket: Optional[str] = None,
    client=None,
) -> dict:
    """Delete a file from DO Spaces.

    Args:
        key: S3 key of the file to delete.
        bucket: Bucket name. Falls back to DO_SPACES_BUCKET.
        client: Pre-configured S3 client (optional).

    Returns:
        dict with 'success', 'key', and 'message'.
    """
    bucket = bucket or os.environ.get("DO_SPACES_BUCKET", "")

    if not bucket:
        return {"success": False, "key": key, "message": "No bucket specified."}

    try:
        if client is None:
            client = get_spaces_client()

        client.delete_object(Bucket=bucket, Key=key)

        return {
            "success": True,
            "key": key,
            "message": f"Deleted {key} from {bucket}",
        }
    except Exception as e:
        return {"success": False, "key": key, "message": f"Delete failed: {str(e)}"}


# ─── CLI Interface ────────────────────────────────────────────────


def main():
    parser = argparse.ArgumentParser(
        description="🦞 Manage files in DO Spaces"
    )
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--upload", metavar="FILE", help="Upload a file")
    group.add_argument("--list", action="store_true", help="List files")
    group.add_argument("--delete", metavar="KEY", help="Delete a file by key")

    parser.add_argument("--bucket", default=None, help="Spaces bucket name")
    parser.add_argument("--prefix", default="", help="Key prefix (folder)")
    parser.add_argument("--key", default=None, help="Custom S3 key for uploads")
    parser.add_argument("--json", action="store_true", help="Output as JSON")

    args = parser.parse_args()

    if args.upload:
        filepath = Path(args.upload)
        if not filepath.exists():
            print(f"Error: File not found: {filepath}", file=sys.stderr)
            sys.exit(1)

        content = filepath.read_text()
        key = args.key or build_key(args.prefix, filepath.name)
        result = upload_file(content, key, bucket=args.bucket)

    elif args.list:
        result = list_files(bucket=args.bucket, prefix=args.prefix)

    elif args.delete:
        result = delete_file(args.delete, bucket=args.bucket)

    if args.json:
        print(json.dumps(result, indent=2, default=str))
    else:
        if result["success"]:
            print(f"✅ {result['message']}")
            if "files" in result:
                for f in result["files"]:
                    print(f"  📄 {f['key']} ({f['size']} bytes)")
        else:
            print(f"❌ {result['message']}", file=sys.stderr)
            sys.exit(1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis:  | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)