Skill: sergical/sentry-observability (version 1.1.0)

--- name: sentry description: "Add observability to your OpenClaw instance — errors, logs, and traces sent to Sentry. Set up monitoring with the Sentry plugin, then investigate issues with the `sentry` CLI." metadata: { "openclaw": { "emoji": "🐛", "requires": { "bins": ["sentry"] }, "install": [ { "id": "npm", "kind": "npm", "package": "sentry", "global": true, "bins": ["sentry"], "label": "Install Sentry CLI (npm)", }, ], }, } --- # Sentry — OpenClaw Observability See what your OpenClaw instance is doing: errors, structured logs, and performance traces — all in Sentry. Two halves: **setup** (get telemetry flowing) and **investigation** (query it with the CLI). --- ## Setup ### 1. Authenticate ```bash sentry auth login ``` OAuth device flow — follow the browser prompt. Credentials stored in `~/.sentry/cli.db`. Alternatives (one-liners): - `sentry auth login --token <TOKEN>` — paste an auth token directly - `SENTRY_AUTH_TOKEN=<token>` — env var, useful in CI ### 2. Create a Project Create a dedicated Sentry project for your OpenClaw instance: ```bash sentry api /teams/<org>/<team>/projects/ \ --method POST \ --field name="my-openclaw" \ --field platform=node ``` Don't know your org/team slugs? List them: ```bash sentry api /organizations/ # list orgs sentry api /organizations/<org>/teams/ # list teams in org ``` ### 3. Get the DSN ```bash sentry project view <org>/my-openclaw --json | jq -r '.dsn' ``` Or via the keys endpoint: ```bash sentry api /projects/<org>/my-openclaw/keys/ | jq '.[0].dsn.public' ``` ### 4. Configure OpenClaw Add the DSN to your `openclaw.json`: ```json { "plugins": { "entries": { "sentry": { "enabled": true, "config": { "dsn": "https://[REDACTED_SECRET]/0", "enableLogs": true } } } } } ``` > **Note:** Config goes under `plugins.entries.sentry.config`, not directly under `sentry`. Then install the Sentry plugin. See `references/plugin-setup.md` for the full plugin implementation using `@sentry/node`. > **Log buffer gotcha:** Sentry's structured logs buffer up to 100 items before auto-flushing. For low-volume services like OpenClaw, logs may sit in the buffer for a long time. The plugin should call `_INTERNAL_flushLogsBuffer(client)` periodically (e.g. every 30s) and before `Sentry.flush()` on shutdown. See `references/plugin-setup.md` for the implementation. ### 5. Verify Restart your OpenClaw gateway, then check Sentry for incoming events: ```bash sentry issue list <org>/my-openclaw --limit 5 ``` --- ## Investigation Once telemetry is flowing, use the CLI to query your OpenClaw's errors, traces, and events. ### List Issues ```bash sentry issue list <org>/<project> sentry issue list <org>/<project> --query "is:unresolved" --sort freq --limit 20 sentry issue list <org>/ # all projects in org ``` ### View an Issue ```bash sentry issue view <short-id> # e.g. MY-OPENCLAW-42 sentry issue view <short-id> --json # structured output ``` ### AI Root Cause Analysis ```bash sentry issue explain <issue-id> # Seer analyzes the root cause sentry issue explain <issue-id> --force # force fresh analysis sentry issue plan <issue-id> # generate a fix plan (run explain first) ``` ### Structured Logs ```bash sentry log list <org>/<project> # last 100 logs sentry log list <org>/<project> --limit 50 # last 50 sentry log list <org>/<project> -q 'level:error' # filter by level sentry log list <org>/<project> -q 'database' # filter by message sentry log list <org>/<project> -f # stream in real-time (2s poll) sentry log list <org>/<project> -f 5 # stream with 5s poll sentry log list <org>/<project> --json # structured output ``` View a specific log entry: ```bash sentry log view <log-id> # 32-char hex ID sentry log view <log-id> --json sentry log view <log-id> --web # open in browser ``` ### Inspect Events ```bash sentry event view <event-id> # full stack trace + context sentry event view <event-id> --json ``` ### Direct API Calls ```bash sentry api /projects/<org>/<project>/issues/ --paginate sentry api /issues/<id>/ --method PUT --field status=resolved sentry api /issues/<id>/ --method PUT --field assignedTo="[REDACTED_SECRET]" ``` ### Workflow: Investigate an Error 1. `sentry issue list <org>/<project> --query "is:unresolved" --sort date --limit 5` 2. `sentry issue view <short-id>` — context, affected users, timeline 3. `sentry issue explain <issue-id>` — AI root cause analysis 4. `sentry issue plan <issue-id>` — concrete fix steps 5. Fix → `sentry api /issues/<id>/ --method PUT --field status=resolved` --- ## Reference - Full CLI commands: `references/cli-commands.md` - Plugin implementation: `references/plugin-setup.md` - CLI docs: https://cli.sentry.dev - Sentry API: https://docs.sentry.io/api/ - Node SDK: https://docs.sentry.io/platforms/javascript/guides/node/

path: references/cli-commands.md
content: # Sentry CLI Command Reference

## Authentication

```bash
sentry auth login --token <TOKEN>     # authenticate with auth token
sentry auth status                     # check current auth
```

## issue

Track and manage Sentry issues.

### issue list

```bash
sentry issue list <org>/<project>      # explicit org/project
sentry issue list <org>/               # all projects in org
sentry issue list <project>            # search project across orgs
sentry issue list                      # auto-detect from DSN/config
```

**Options:**
| Option | Description |
|--------|-------------|
| `--query <query>` | Sentry search syntax (e.g., `"is:unresolved TypeError"`) |
| `--sort <sort>` | Sort by: `date` (default), `new`, `freq`, `user` |
| `--limit <n>` | Max results (default: 10) |
| `--json` | JSON output |

**Search query examples:**
- `--query "is:unresolved"` — only unresolved
- `--query "is:resolved"` — only resolved
- `--query "is:unresolved TypeError"` — unresolved + keyword
- `--query "assigned:me"` — assigned to current user

### issue view

```bash
sentry issue view <issue-id>           # by numeric ID
sentry issue view <short-id>           # by short ID (e.g., PROJ-ABC)
sentry issue view <short-id> -w        # open in browser
sentry issue view <id> --json          # JSON output
```

### issue explain

AI-powered root cause analysis using Seer.

```bash
sentry issue explain <issue-id>
sentry issue explain <short-id>
sentry issue explain my-org/MYPROJECT-ABC
sentry issue explain <issue-id> --force   # force fresh analysis
sentry issue explain <issue-id> --json
```

**Requirements:** Seer AI enabled, GitHub integration configured, code mappings set up.

**Output includes:** identified root causes, reproduction steps, relevant code locations.

### issue plan

Generate a fix plan (requires `explain` to have run first).

```bash
sentry issue plan <issue-id>
sentry issue plan <issue-id> --cause 0       # specific root cause
sentry issue plan my-org/MYPROJECT-ABC --cause 1
sentry issue plan <issue-id> --json
```

## event

Inspect Sentry events.

### event view

```bash
sentry event view <event-id>
sentry event view <event-id> -w        # open in browser
sentry event view <event-id> --json
```

**Output includes:** exception type/message, full stack trace, tags (browser, OS, environment, release), context (URL, user_id).

**Finding event IDs:** from `sentry issue view` output, from Sentry UI, or from error reports (`event_id` field).

## project

Manage Sentry projects.

### project list

```bash
sentry project list                    # all accessible projects
sentry project list <org-slug>         # projects in specific org
sentry project list --platform javascript  # filter by platform
sentry project list --json
```

### project view

```bash
sentry project view                    # auto-detect
sentry project view <org>/<project>    # explicit
sentry project view <project>          # search across orgs
sentry project view <target> -w        # open in browser
sentry project view <target> --json
```

**Output includes:** project name, org, platform, team, DSN.

## api

Make direct REST API calls to Sentry.

```bash
sentry api <endpoint> [options]
```

**Options:**
| Option | Description |
|--------|-------------|
| `--method <method>` | HTTP method: GET (default), POST, PUT, DELETE |
| `--field <key=value>` | Request body field (repeatable) |
| `--header <key:value>` | Custom header (repeatable) |
| `--include` | Include response headers |
| `--paginate` | Auto-paginate through all results |

### Common API endpoints

```bash
# Organizations
sentry api /organizations/
sentry api /organizations/my-org/

# Projects
sentry api /projects/my-org/my-project/

# Issues
sentry api /projects/my-org/my-project/issues/ --paginate

# Resolve an issue
sentry api /issues/<id>/ --method PUT --field status=resolved

# Assign an issue
sentry api /issues/<id>/ --method PUT --field assignedTo="[REDACTED_SECRET]"

# Create a project
sentry api /teams/my-org/my-team/projects/ \
  --method POST \
  --field name="New Project" \
  --field platform=javascript

# Delete a project
sentry api /projects/my-org/my-project/ --method DELETE
```

Full API docs: https://docs.sentry.io/api/


path: references/plugin-setup.md
content: # OpenClaw Sentry Plugin — Implementation Guide

How to build an OpenClaw plugin that sends errors, structured logs, and performance traces to Sentry using `@sentry/node`.

---

## Overview

The plugin hooks into three OpenClaw mechanisms:

| Mechanism | What it captures |
|---|---|
| `@sentry/node` init | Unhandled errors, process crashes |
| `onDiagnosticEvent()` | Performance traces — LLM calls, message processing, webhook errors |
| `registerLogTransport()` | Structured logs from the gateway |

No OpenTelemetry dependencies needed. The Sentry Node SDK handles everything.

---

## Install Dependencies

```bash
npm install @sentry/node @sentry/core
```

> `@sentry/core` is needed for `_INTERNAL_flushLogsBuffer` — Sentry's log buffer doesn't auto-flush until 100 items accumulate, which low-volume services may never hit.

---

## Plugin Code

See the full implementation at https://github.com/sergical/openclaw-plugin-sentry

Key implementation details:

### Diagnostic Events → Sentry Spans

```typescript
import * as Sentry from "@sentry/node";

// model.usage → ai.chat span with GenAI semantic conventions
function recordModelUsage(evt) {
  const endTimeMs = evt.ts;
  const durationMs = evt.durationMs ?? 100;
  const startTimeMs = endTimeMs - durationMs;

  const span = Sentry.startInactiveSpan({
    op: "ai.chat",
    name: `chat ${evt.model}`,
    startTime: startTimeMs,       // Sentry v10 accepts ms (NOT seconds!)
    forceTransaction: true,
    attributes: {
      "gen_ai.operation.name": "chat",
      "gen_ai.system": evt.provider ?? "unknown",
      "gen_ai.request.model": evt.model ?? "unknown",
      "gen_ai.usage.input_tokens": evt.usage.input ?? 0,
      "gen_ai.usage.output_tokens": evt.usage.output ?? 0,
    },
  });
  span?.end(endTimeMs);
}
```

### Log Forwarding with Buffer Flush

```typescript
import { _INTERNAL_flushLogsBuffer } from "@sentry/core";

// In start():
// Sentry buffers logs until 100 items. Flush every 30s for low-volume instances.
const logFlushInterval = setInterval(() => {
  const client = Sentry.getClient();
  if (client) _INTERNAL_flushLogsBuffer(client);
}, 30_000);

// In stop():
clearInterval(logFlushInterval);
// Flush log buffer BEFORE flushing transport — otherwise buffered logs are lost
const client = Sentry.getClient();
if (client) _INTERNAL_flushLogsBuffer(client);
await Sentry.flush(5000);
```

### Log Transport Format

OpenClaw's log transport uses numeric keys as positional args:
- Key `"0"` = subsystem tag (e.g. `{"subsystem":"plugins"}`)
- Key `"1"` = message string
- `_meta` has `logLevelName`, `name`, `date`

```typescript
function forwardLog(logObj: Record<string, unknown>): void {
  const meta = logObj._meta as { logLevelName?: string } | undefined;
  const level = (meta?.logLevelName ?? "INFO").toLowerCase();

  // Extract subsystem and message from positional args
  const numericEntries = Object.entries(logObj)
    .filter(([key]) => /^\d+$/.test(key))
    .sort((a, b) => Number(a[0]) - Number(b[0]));

  let subsystem = "";
  let message = "";

  if (numericEntries.length >= 2) {
    subsystem = String(numericEntries[0][1]);
    message = String(numericEntries[numericEntries.length - 1][1]);
  } else if (numericEntries.length === 1) {
    message = String(numericEntries[0][1]);
  }

  const attrs: Record<string, string> = {};
  if (subsystem) attrs["openclaw.subsystem"] = subsystem;

  // Route to Sentry.logger at appropriate level
  switch (level) {
    case "debug": case "trace": Sentry.logger.debug(message, attrs); break;
    case "warn": Sentry.logger.warn(message, attrs); break;
    case "error": case "fatal": Sentry.logger.error(message, attrs); break;
    default: Sentry.logger.info(message, attrs);
  }
}
```

---

## Configuration

Add to `openclaw.json` — note config goes under `plugins.entries.sentry.config`:

```json
{
  "plugins": {
    "entries": {
      "sentry": {
        "enabled": true,
        "config": {
          "dsn": "https://[REDACTED_SECRET]/0",
          "environment": "production",
          "tracesSampleRate": 1.0,
          "enableLogs": true
        }
      }
    }
  }
}
```

| Field | Required | Description |
|---|---|---|
| `dsn` | **yes** | Sentry DSN from your project's Client Keys |
| `environment` | no | `"production"`, `"staging"`, etc. Default: `"production"` |
| `tracesSampleRate` | no | `0.0` – `1.0`. Default: `1.0` (sample everything) |
| `enableLogs` | no | Enable structured log forwarding. Default: `true` |

---

## Gotchas

### Sentry v10 Timestamps
`startTime` and `span.end()` accept **milliseconds** (or Date objects or `[sec, ns]` tuples). Do NOT divide by 1000 — that creates 0ms duration spans.

### Log Buffer
Sentry's structured logs buffer internally (max 100 items). `Sentry.flush()` does NOT flush this buffer. You must call `_INTERNAL_flushLogsBuffer(client)` from `@sentry/core` explicitly, then `Sentry.flush()` to send the resulting envelope.

### Plugin Config Path
Config must be at `plugins.entries.sentry.config.dsn`, not `plugins.entries.sentry.dsn`. The plugin's `configSchema` needs `additionalProperties: true`.

### Module Isolation
OpenClaw's bundler creates separate chunks for gateway and plugin-sdk, so `onDiagnosticEvent` and `registerLogTransport` may use different listener Sets. If events aren't reaching the plugin, check if the `globalThis.__oc_diag` / `globalThis.__oc_log_transports` patches are applied.

---

## What You'll See in Sentry

### Errors
Unhandled exceptions, webhook errors, and session stuck warnings.

### Traces
- `ai.chat` spans with GenAI semantic conventions (`gen_ai.request.model`, `gen_ai.usage.*`)
- `openclaw.message` spans for message processing with duration

### Logs
Structured gateway logs with level mapping and `openclaw.subsystem` attributes. View with:

```bash
sentry log list <org>/<project>
sentry log list <org>/<project> -f    # stream in real-time
```

---

## Docs

- Sentry Node SDK: https://docs.sentry.io/platforms/javascript/guides/node/
- Sentry Structured Logs: https://docs.sentry.io/platforms/javascript/guides/node/logs/
- Sentry CLI Logs: https://cli.sentry.dev/commands/log/
- Plugin source: https://github.com/sergical/openclaw-plugin-sentry


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)