Skill: nissan/observability-lgtm (version 1.2.2)

--- name: observability-lgtm version: 1.2.2 description: Set up a full local LGTM observability stack (Loki + Grafana + Tempo + Prometheus + Alloy) for FastAPI apps. One Docker Compose, one Python import, unified dashboards. homepage: https://github.com/reddinft/skill-observability-lgtm metadata: { "openclaw": { "emoji": "📊", "requires": { "bins": ["docker", "docker-compose"], "env": [] }, "network": { "outbound": false, "reason": "All services run locally in Docker. No external network calls. Grafana/Prometheus/Loki/Tempo are self-hosted only." } } } --- **Last used:** 2026-03-24 **Memory references:** 7 **Status:** Active # observability-lgtm Set up a full local observability stack (Loki + Grafana + Tempo + Prometheus + Alloy) for FastAPI apps on macOS (Apple Silicon) or Linux. One command to start, one import to instrument any app. Logs → Loki, metrics → Prometheus, traces → Tempo, all unified in Grafana. ## When to use - User is building a FastAPI web app and wants logs, metrics, and traces - User wants a local Grafana dashboard without setting up ELK (too heavy) - User wants to correlate logs ↔ traces ↔ metrics in one UI - User has multiple local apps and wants universal observability ## When NOT to use - Production cloud deployments (use managed Grafana Cloud or Datadog instead) - Non-Python apps (the Python lib only works for FastAPI; the stack itself is language-agnostic) - When Docker is not available ## Prerequisites - Docker + Docker Compose v2 installed - Python 3.10+ (for the instrumentation lib) - FastAPI app to instrument ## What gets installed | Service | Port | Purpose | |---|---|---| | Grafana | 3000 | Dashboards — no login in dev mode | | Prometheus | 9091 | Metrics scraping (avoids 9090 if MinIO running) | | Loki | 3300 | Log storage (avoids 3100 if Langfuse running) | | Tempo gRPC | 4317 | OTLP trace receiver | | Tempo HTTP | 4318 | OTLP HTTP alternative | | Alloy UI | 12345 | Agent status | ## Steps ### Step 1 — Check for port conflicts ```bash lsof -iTCP -sTCP:LISTEN -n -P 2>/dev/null | grep -E ":(3000|3300|9091|4317|4318|12345)" | awk '{print $9, $1}' ``` If any of the ports above are in use, update the relevant port in docker-compose.yml and the matching `url:` in config/grafana/provisioning/datasources/datasources.yml. Common conflicts: Langfuse on 3100, MinIO on 9090. ### Step 2 — Copy the stack Copy these files from the skill directory into a `projects/observability/` folder in the workspace: - `assets/docker-compose.yml` - `assets/config/` (entire directory tree) - `assets/lib/observability.py` - `assets/scripts/register_app.sh` ```bash mkdir -p projects/observability cp -r SKILL_DIR/assets/* projects/observability/ mkdir -p projects/observability/logs touch projects/observability/logs/.gitkeep chmod +x projects/observability/scripts/register_app.sh ``` ### Step 3 — Start the stack ```bash cd projects/observability docker compose up -d ``` Wait ~15 seconds for all services to start, then verify: ```bash curl -s -o /dev/null -w "Grafana: %{http_code}\n" http://localhost:3000/api/health curl -s -o /dev/null -w "Prometheus: %{http_code}\n" http://localhost:9091/-/healthy curl -s -o /dev/null -w "Loki: %{http_code}\n" http://localhost:3300/ready curl -s -o /dev/null -w "Tempo: %{http_code}\n" http://localhost:4318/ready ``` All should return 200. If Loki or Tempo return 503, wait 10 more seconds and retry (they have a slower startup than Grafana/Prometheus). ### Step 4 — Install Python deps for the app ```bash pip install \ "prometheus-fastapi-instrumentator>=7.0.0" \ "opentelemetry-sdk>=1.25.0" \ "opentelemetry-exporter-otlp-proto-grpc>=1.25.0" \ "opentelemetry-instrumentation-fastapi>=0.46b0" \ "python-json-logger>=2.0.7" ``` ### Step 5 — Instrument the FastAPI app Add to the app's `app.py` (or `main.py`), just after `app = FastAPI(...)`: ```python import sys sys.path.insert(0, [REDACTED_SECRET]) from observability import setup_observability logger = setup_observability(app, service_name="my-service-name") ``` That's it. The app now: - Exposes `/metrics` for Prometheus - Writes JSON logs to `projects/observability/logs/my-service-name/app.log` - Sends traces to Tempo on localhost:4317 ### Step 6 — Register with Prometheus ```bash cd projects/observability ./scripts/register_app.sh my-service-name <port> # e.g.: ./scripts/register_app.sh image-gen-studio 7860 ``` Prometheus hot-reloads the target within 30 seconds. Verify: ```bash curl -s "http://localhost:9091/api/v1/targets" | python3 -c " import json, sys data = json.load(sys.stdin) for t in data['data']['activeTargets']: svc = t['labels'].get('service', '') print(svc, '->', t['health']) " ``` ### Step 7 — Open Grafana Open http://localhost:3000 The **FastAPI — App Overview** dashboard is pre-loaded. Select your service from the dropdown at the top. You'll see: - Request rate (req/s) - Error rate (%) - Latency p50/p95/p99 - Requests by endpoint - HTTP status codes - Live log panel (Loki) To jump from a log line to its trace: click the trace_id link in the log detail panel. It opens the full trace in Tempo automatically (datasource pre-wired). ### Step 8 — Import additional dashboards (optional) In Grafana → Dashboards → Import: - **16110** — FastAPI Observability (richer alternative to the built-in) - **13407** — Loki Logs Overview - **16112** — Tempo Service Graph (service dependency map) ## Useful commands ```bash # Reload Prometheus config after registering a new app: curl -s -X POST http://localhost:9091/-/reload # Restart a single service without losing data: docker compose -f projects/observability/docker-compose.yml restart grafana # Stop everything (data volumes preserved): docker compose -f projects/observability/docker-compose.yml down # Nuclear reset (wipes all stored data): docker compose -f projects/observability/docker-compose.yml down -v # Check Alloy log shipping status: open http://localhost:12345 ``` ## Manual tracing (optional) ```python from observability import get_tracer tracer = get_tracer(__name__) @app.get("/expensive-endpoint") async def handler(): with tracer.start_as_current_span("db-query") as span: span.set_attribute("db.table", "users") result = await db.query(...) return result ``` ## Log/trace correlation The OTel instrumentation injects `trace_id` into every log record. Grafana Loki is pre-configured with a derived field that turns `"trace_id":"abc123"` into a clickable link to the Tempo trace. To manually include trace context in your own log calls: ```python from opentelemetry import trace def trace_ctx() -> dict: ctx = trace.get_current_span().get_span_context() return {"trace_id": format(ctx.trace_id, "032x")} if ctx.is_valid else {} logger.info("Processing request", extra=trace_ctx()) ``` ## Notes - Logs are written to `projects/observability/logs/<service>/app.log` as JSON. Alloy tails these files and ships to Loki — no code changes needed beyond setup_observability(). - All observability is local — no data leaves the machine. - `data_classification: LOCAL_ONLY` is the default for all traces/logs. - The Alloy config drops DEBUG-level logs by default. Edit `config/alloy/config.alloy` to remove the `stage.drop` block if you need debug logs.

path: assets/config/grafana/dashboards/fastapi-overview.json
content: {
  "__inputs": [
    { "name": "DS_PROMETHEUS", "label": "Prometheus", "type": "datasource", "pluginId": "prometheus" },
    { "name": "DS_LOKI",       "label": "Loki",       "type": "datasource", "pluginId": "loki" },
    { "name": "DS_TEMPO",      "label": "Tempo",      "type": "datasource", "pluginId": "tempo" }
  ],
  "__requires": [],
  "title": "FastAPI — App Overview",
  "uid": "fastapi-overview",
  "description": "Request rate, error rate, latency, logs. Select service from the dropdown.",
  "tags": ["fastapi", "openclaw"],
  "schemaVersion": 38,
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-1h", "to": "now" },
  "timepicker": {},
  "templating": {
    "list": [
      {
        "name": "service",
        "label": "Service",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus" },
        "query": "label_values(http_requests_total, service)",
        "sort": 1,
        "includeAll": true,
        "allValue": ".*",
        "current": {},
        "refresh": 2
      },
      {
        "name": "interval",
        "label": "Interval",
        "type": "interval",
        "query": "30s,1m,5m,10m",
        "current": { "value": "1m" }
      }
    ]
  },
  "panels": [
    {
      "id": 1,
      "title": "Request Rate (req/s)",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": {
        "defaults": {
          "color": { "mode": "palette-classic" },
          "custom": { "lineWidth": 2 },
          "unit": "reqps"
        }
      },
      "options": { "legend": { "displayMode": "list", "placement": "bottom" } },
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service)",
          "legendFormat": "{{service}}",
          "refId": "A"
        }
      ]
    },
    {
      "id": 2,
      "title": "Error Rate (%)",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0, "max": 100,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 1 },
              { "color": "red", "value": 5 }
            ]
          },
          "custom": { "lineWidth": 2 }
        }
      },
      "targets": [
        {
          "expr": "100 * sum(rate(http_requests_total{service=~\"$service\", status=~\"5..\"}[$interval])) by (service) / sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service)",
          "legendFormat": "{{service}} 5xx%",
          "refId": "A"
        }
      ]
    },
    {
      "id": 3,
      "title": "Latency (p50 / p95 / p99)",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": {
        "defaults": { "unit": "s", "custom": { "lineWidth": 2 } }
      },
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (le, service))",
          "legendFormat": "{{service}} p50", "refId": "A"
        },
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (le, service))",
          "legendFormat": "{{service}} p95", "refId": "B"
        },
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (le, service))",
          "legendFormat": "{{service}} p99", "refId": "C"
        }
      ]
    },
    {
      "id": 4,
      "title": "Requests by Endpoint",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": {
        "defaults": { "unit": "reqps", "custom": { "lineWidth": 1 } }
      },
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (handler, method)",
          "legendFormat": "{{method}} {{handler}}", "refId": "A"
        }
      ]
    },
    {
      "id": 5,
      "title": "HTTP Status Codes",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": {
        "defaults": { "unit": "reqps" },
        "overrides": [
          { "matcher": { "id": "byName", "options": "2xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] },
          { "matcher": { "id": "byName", "options": "4xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "yellow", "mode": "fixed" } }] },
          { "matcher": { "id": "byName", "options": "5xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] }
        ]
      },
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{service=~\"$service\", status=~\"2..\"}[$interval]))",
          "legendFormat": "2xx", "refId": "A"
        },
        {
          "expr": "sum(rate(http_requests_total{service=~\"$service\", status=~\"4..\"}[$interval]))",
          "legendFormat": "4xx", "refId": "B"
        },
        {
          "expr": "sum(rate(http_requests_total{service=~\"$service\", status=~\"5..\"}[$interval]))",
          "legendFormat": "5xx", "refId": "C"
        }
      ]
    },
    {
      "id": 6,
      "title": "Logs",
      "type": "logs",
      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 16 },
      "datasource": { "type": "loki", "uid": "loki" },
      "options": {
        "dedupStrategy": "none",
        "enableLogDetails": true,
        "prettifyLogMessage": true,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": true
      },
      "targets": [
        {
          "expr": "{service=~\"$service\"}",
          "refId": "A"
        }
      ]
    }
  ]
}


path: assets/config/grafana/provisioning/dashboards/dashboards.yml
content: apiVersion: 1

providers:
  - name: openclaw
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true


path: assets/config/grafana/provisioning/datasources/datasources.yml
content: apiVersion: 1

datasources:

  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: false
    jsonData:
      httpMethod: POST
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo

  - name: Loki
    type: loki
    uid: loki
    url: http://loki:3100
    access: proxy
    isDefault: true
    jsonData:
      derivedFields:
        - datasourceUid: tempo
          matcherRegex: '"trace_id":"(\w+)"'
          name: trace_id
          url: "$${__value.raw}"   # link to Tempo trace

  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo:3200
    access: proxy
    isDefault: false
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: prometheus
      nodeGraph:
        enabled: true
      traceQuery:
        timeShiftEnabled: true
        spanStartTimeShift: -30m
        spanEndTimeShift: 30m
      lokiSearch:
        datasourceUid: loki
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: -5m
        spanEndTimeShift: 5m
        tags:
          - key: service.name
            value: service
      tracesToMetrics:
        datasourceUid: prometheus


path: assets/config/loki/loki.yml
content: auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: warn

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_length: 721h    # 30 days
  retention_period: 720h    # 30 days

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem


path: assets/config/prometheus/prometheus.yml
content: global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: local-mac
    env: dev

# ── Scrape targets ────────────────────────────────────────────────────────────

scrape_configs:

  # Self-monitoring
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]

  # Grafana
  - job_name: grafana
    static_configs:
      - targets: [grafana:3000]

  # Loki
  - job_name: loki
    static_configs:
      - targets: [loki:3100]

  # Tempo
  - job_name: tempo
    static_configs:
      - targets: [tempo:3200]

  # Alloy
  - job_name: alloy
    static_configs:
      - targets: [alloy:12345]

  # ── Native Mac apps (via file-based service discovery) ────────────────────
  # Add apps with: scripts/register_app.sh <name> <host_port>
  # Prometheus reaches the Mac host via host.docker.internal
  - job_name: openclaw_apps
    file_sd_configs:
      - files:
          - /etc/prometheus/targets/*.json
        refresh_interval: 30s
    relabel_configs:
      # Replace host.docker.internal with the actual host
      - source_labels: [__address__]
        target_label: __address__

  # Tempo-generated service graph metrics (RED metrics per service pair)
  - job_name: tempo_metrics_generator
    static_configs:
      - targets: [tempo:3200]
    metrics_path: /metrics


path: assets/config/tempo/tempo.yml
content: server:
  http_listen_port: 3200
  log_level: warn

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 720h   # 30 days

metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

query_frontend:
  search:
    duration_slo: 5s
    throughput_bytes_slo: 1.073741824e+09
  trace_by_id:
    duration_slo: 5s

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]


path: assets/docker-compose.yml
content: # Observability Stack — LGTM (Loki + Grafana + Tempo + Prometheus) + Alloy
# ─────────────────────────────────────────────────────────────────────────────
# Port map (avoiding conflicts with existing services):
#   Grafana       → localhost:3000
#   Prometheus    → localhost:9091  (9090 is MinIO)
#   Loki          → localhost:3300  (3100 is Langfuse)
#   Tempo gRPC    → localhost:4317  (OTLP from native apps)
#   Tempo HTTP    → localhost:4318  (OTLP HTTP from native apps)
#   Alloy UI      → localhost:12345
# ─────────────────────────────────────────────────────────────────────────────

name: observability

networks:
  obs:
    driver: bridge

volumes:
  grafana_data:
  prometheus_data:
  loki_data:
  tempo_data:

services:

  # ── Grafana — unified dashboard ───────────────────────────────────────────
  grafana:
    image: grafana/grafana-oss:11.4.0
    container_name: obs-grafana
    restart: unless-stopped
    networks: [obs]
    ports:
      - "3000:3000"
    environment:
      # No-auth for local dev
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
      GF_FEATURE_TOGGLES_ENABLE: "traceqlEditor tempoSearch"
      GF_SERVER_ROOT_URL: "http://localhost:3000"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./config/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./config/grafana/dashboards:/var/lib/grafana/dashboards:ro
    depends_on: [loki, prometheus, tempo]

  # ── Prometheus — metrics ──────────────────────────────────────────────────
  prometheus:
    image: prom/prometheus:v3.1.0
    container_name: obs-prometheus
    restart: unless-stopped
    networks: [obs]
    ports:
      - "9091:9090"
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle          # hot-reload: curl -X POST localhost:9091/-/reload
      - --web.enable-remote-write-receiver
    volumes:
      - prometheus_data:/prometheus
      - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./config/prometheus/targets:/etc/prometheus/targets:ro  # file-based discovery
    extra_hosts:
      - "host.docker.internal:host-gateway"   # reach native Mac apps

  # ── Loki — logs ──────────────────────────────────────────────────────────
  loki:
    image: grafana/loki:3.3.2
    container_name: obs-loki
    restart: unless-stopped
    networks: [obs]
    ports:
      - "3300:3100"
    command: -config.file=/etc/loki/loki.yml
    volumes:
      - loki_data:/loki
      - ./config/loki/loki.yml:/etc/loki/loki.yml:ro

  # ── Tempo — distributed traces ────────────────────────────────────────────
  tempo:
    image: grafana/tempo:2.7.1
    container_name: obs-tempo
    restart: unless-stopped
    networks: [obs]
    ports:
      - "4317:4317"   # OTLP gRPC (apps send traces here)
      - "4318:4318"   # OTLP HTTP
      - "3400:3200"   # Tempo HTTP query API
    command: -config.file=/etc/tempo/tempo.yml
    volumes:
      - tempo_data:/var/tempo
      - ./config/tempo/tempo.yml:/etc/tempo/tempo.yml:ro

  # ── Grafana Alloy — unified agent (log shipping + OTLP pipeline) ──────────
  alloy:
    image: grafana/alloy:v1.5.1
    container_name: obs-alloy
    restart: unless-stopped
    networks: [obs]
    ports:
      - "12345:12345"   # Alloy UI
    command:
      - run
      - /etc/alloy/config.alloy
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
    volumes:
      - ./config/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - ./logs:/var/log/openclaw:ro   # app log files (read-only)
    extra_hosts:
      - "host.docker.internal:host-gateway"


path: assets/lib/observability.py
content: """
openclaw observability — drop-in for FastAPI apps.

Usage:
    from observability import setup_observability
    setup_observability(app, service_name="my-service")

What you get:
    • /metrics          — Prometheus endpoint (auto-scraped)
    • Structured JSON logs → Loki (via Alloy log scraping)
    • OpenTelemetry traces → Tempo (OTLP gRPC on localhost:4317)
    • Langfuse integration preserved (existing Langfuse usage unaffected)

Deps (add to requirements.txt):
    prometheus-fastapi-instrumentator>=7.0.0
    opentelemetry-sdk>=1.25.0
    opentelemetry-exporter-otlp-proto-grpc>=1.25.0
    opentelemetry-instrumentation-fastapi>=0.46b0
    python-json-logger>=2.0.7
"""

from __future__ import annotations

import logging
import os
import sys
from pathlib import Path
from typing import Callable

# ── Optional imports (graceful degradation) ───────────────────────────────────
try:
    from prometheus_fastapi_instrumentator import Instrumentator
    _HAS_PROM = True
except ImportError:
    _HAS_PROM = False

try:
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    from opentelemetry.sdk.resources import Resource, SERVICE_NAME
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
    from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
    _HAS_OTEL = True
except ImportError:
    _HAS_OTEL = False

try:
    from pythonjsonlogger import jsonlogger
    _HAS_JSONLOG = True
except ImportError:
    _HAS_JSONLOG = False


# ── Log directory ─────────────────────────────────────────────────────────────
_LOG_BASE = Path(os.environ.get(
    "OPENCLAW_LOG_DIR",
    Path(__file__).parent.parent / "logs",   # relative to observability/
))


def _setup_logging(service_name: str, log_level: int = logging.INFO) -> logging.Logger:
    """
    Configure root logger with:
      - JSON formatter → file  (Alloy scrapes this → Loki)
      - Plain formatter → stderr (local dev readability)

    Log file: <OPENCLAW_LOG_DIR>/<service_name>/app.log
    """
    log_dir = _LOG_BASE / service_name
    log_dir.mkdir(parents=True, exist_ok=True)
    log_file = log_dir / "app.log"

    root = logging.getLogger()
    root.setLevel(log_level)

    # Avoid adding duplicate handlers on re-import
    if any(isinstance(h, logging.FileHandler) and str(log_file) in str(getattr(h, 'baseFilename', ''))
           for h in root.handlers):
        return logging.getLogger(service_name)

    # JSON handler → file → Alloy → Loki
    file_handler = logging.FileHandler(log_file, encoding="utf-8")
    if _HAS_JSONLOG:
        fmt = jsonlogger.JsonFormatter(
            fmt="%(asctime)s %(levelname)s %(name)s %(message)s",
            datefmt="%Y-%m-%dT%H:%M:%S",
            rename_fields={"asctime": "ts", "levelname": "level", "name": "logger"},
        )
    else:
        # Fallback: manual JSON-ish format (still parseable by Loki)
        fmt = logging.Formatter(
            '{"ts":"%(asctime)s","level":"%(levelname)s","logger":"%(name)s","service":"' + service_name + '","msg":"%(message)s"}',
            datefmt="%Y-%m-%dT%H:%M:%S",
        )
    file_handler.setFormatter(fmt)
    file_handler.setLevel(log_level)
    root.addHandler(file_handler)

    # Plain stderr handler for local dev
    stderr_handler = logging.StreamHandler(sys.stderr)
    stderr_handler.setFormatter(logging.Formatter(
        "%(asctime)s  %(levelname)-8s  %(name)s  %(message)s",
        datefmt="%H:%M:%S",
    ))
    stderr_handler.setLevel(log_level)
    root.addHandler(stderr_handler)

    logger = logging.getLogger(service_name)
    logger.info("Logging initialised", extra={"service": service_name, "log_file": str(log_file)})
    return logger


def _setup_tracing(service_name: str, otlp_endpoint: str) -> None:
    """Configure OpenTelemetry traces → Tempo via OTLP gRPC."""
    if not _HAS_OTEL:
        return

    resource = Resource.create({SERVICE_NAME: service_name})
    provider = TracerProvider(resource=resource)

    exporter = OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True)
    provider.add_span_processor(BatchSpanProcessor(exporter))

    trace.set_tracer_provider(provider)


def _setup_metrics(app, service_name: str) -> None:
    """Mount Prometheus /metrics endpoint and auto-instrument all FastAPI routes."""
    if not _HAS_PROM:
        return

    Instrumentator(
        should_group_status_codes=True,
        should_ignore_untemplated=True,
        excluded_handlers=[r"/metrics", r"/health", r"/static/.*"],
        env_var_name=None,
    ).instrument(app).expose(app, endpoint="/metrics", include_in_schema=False)


def setup_observability(
    app,
    service_name: str,
    *,
    log_level: int = logging.INFO,
    otlp_endpoint: str | None = None,
    enable_traces: bool = True,
    enable_metrics: bool = True,
    enable_logs: bool = True,
) -> logging.Logger:
    """
    One-call setup for all observability pillars.

    Args:
        app:            FastAPI application instance.
        service_name:   Identifier for this service (used in all telemetry labels).
        log_level:      Python logging level. Default INFO.
        otlp_endpoint:  Override OTLP gRPC endpoint. Default: localhost:4317.
        enable_traces:  Send traces to Tempo. Default True.
        enable_metrics: Expose /metrics for Prometheus. Default True.
        enable_logs:    Write JSON logs to file for Alloy→Loki. Default True.

    Returns:
        A configured logger for the service.
    """
    if otlp_endpoint is None:
        otlp_endpoint = os.environ.get("OTLP_ENDPOINT", "localhost:4317")

    logger = logging.getLogger(service_name)

    if enable_logs:
        logger = _setup_logging(service_name, log_level)

    if enable_metrics:
        _setup_metrics(app, service_name)

    if enable_traces:
        _setup_tracing(service_name, otlp_endpoint)
        if _HAS_OTEL:
            FastAPIInstrumentor.instrument_app(
                app,
                tracer_provider=trace.get_tracer_provider(),
                excluded_urls="/metrics,/health,/static/.*",
            )

    logger.info(
        "Observability initialised",
        extra={
            "service": service_name,
            "metrics": enable_metrics and _HAS_PROM,
            "traces": enable_traces and _HAS_OTEL,
            "logs": enable_logs,
            "otlp": otlp_endpoint,
        },
    )
    return logger


# ── Convenience: get a tracer for manual instrumentation ─────────────────────

def get_tracer(name: str = __name__):
    """Get an OpenTelemetry tracer. Call setup_observability() first."""
    if not _HAS_OTEL:
        return _NoopTracer()
    return trace.get_tracer(name)


class _NoopTracer:
    """Fallback when OpenTelemetry is not installed."""
    def start_as_current_span(self, *a, **kw):
        from contextlib import contextmanager
        @contextmanager
        def _noop(*_, **__):
            yield self
        return _noop()
    def set_attribute(self, *a, **kw): pass
    def record_exception(self, *a, **kw): pass


path: assets/requirements.txt
content: # Add these to any FastAPI app that uses observability.py
# Install: pip install -r requirements.txt

# Metrics
prometheus-fastapi-instrumentator>=7.0.0

# Traces (OpenTelemetry)
opentelemetry-sdk>=1.25.0
opentelemetry-exporter-otlp-proto-grpc>=1.25.0
opentelemetry-instrumentation-fastapi>=0.46b0

# Structured JSON logging (Loki-compatible)
python-json-logger>=2.0.7


path: assets/scripts/register_app.sh
content: #!/usr/bin/env bash
# register_app.sh <service_name> <port>
# Adds a native Mac app to Prometheus file-based service discovery.
# Prometheus hot-reloads the target within 30 seconds (no restart needed).
#
# Usage:
#   ./scripts/register_app.sh image-gen-studio 7860
#   ./scripts/register_app.sh hybrid-control-plane 8765

set -euo pipefail

SERVICE="${1:?Usage: $0 <service_name> <port>}"
PORT="${2:?Usage: $0 <service_name> <port>}"

TARGETS_DIR="$(cd "$(dirname "$0")/../config/prometheus/targets" && pwd)"
TARGET_FILE="${TARGETS_DIR}/${SERVICE}.json"

cat > "${TARGET_FILE}" <<JSON
[
  {
    "targets": ["host.docker.internal:${PORT}"],
    "labels": {
      "job": "${SERVICE}",
      "service": "${SERVICE}",
      "env": "dev"
    }
  }
]
JSON

echo "✅ Registered: ${SERVICE} → host.docker.internal:${PORT}"
echo "   Target file: ${TARGET_FILE}"
echo "   Prometheus will pick this up within 30 seconds."
echo ""
echo "   Verify:  curl -s localhost:9091/api/v1/targets | jq '.data.activeTargets[] | select(.labels.service == \"${SERVICE}\")'"


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal:  | SkillSpector: suspicious (MEDIUM)