Skill: matthewengman/evalpal (version 1.0.1)

--- name: evalpal description: Run AI agent evaluations via EvalPal — trigger eval runs, check results, and list available evaluations homepage: https://evalpal.dev user-invocable: true requires: env: - EVALPAL_API_KEY bins: - curl - jq --- # EvalPal Skill Run AI agent evaluations inline. Trigger eval runs, poll for results, and list available evaluation definitions — all from chat. ## Prerequisites Set the following environment variables in your OpenClaw skill configuration: | Variable | Required | Description | | ----------------- | -------- | -------------------------------------------- | | `EVALPAL_API_KEY` | Yes | Your EvalPal API key (starts with `sk_`) | | `EVALPAL_API_URL` | No | Base URL (defaults to `https://evalpal.dev`) | Get your API key from **Settings → API Keys** at [evalpal.dev](https://evalpal.dev). ## Commands ### `/evalpal run --eval-id <ID>` Trigger an evaluation run and wait for results. **Usage:** ```bash bash scripts/run-eval.sh --eval-id <EVAL_DEFINITION_ID> ``` **What it does:** 1. Triggers a new eval run via the EvalPal API 2. Polls for completion with exponential backoff (up to 5 minutes) 3. Fetches and formats results as readable markdown **Example output:** ``` ✅ Episode Quality — PASSED (15/16) ├── Test Case tc_001: ✓ PASS ├── Test Case tc_002: ✓ PASS ├── Test Case tc_003: ✗ FAIL └── 12 more passed... Run ID: run_abc123 · 16 test cases · 47s ``` **Exit codes:** 0 = all passed, 1 = failures or error. ### `/evalpal status --run-id <ID>` Check the current status of a running evaluation. **Usage:** ```bash bash scripts/check-status.sh --run-id <RUN_ID> ``` **Example output:** ``` 📊 Run Status: run_abc123 Status: running Started: 2026-03-26T20:00:00Z ``` ### `/evalpal list` List available evaluation definitions across your projects. **Usage:** ```bash bash scripts/list-evals.sh [--project-id <PROJECT_ID>] ``` If `--project-id` is omitted, lists evals for all projects. **Example output:** ``` 📋 Evaluation Definitions Project: AI Workforce Lab abc123 Episode Quality Check def456 Factual Accuracy Eval Project: Customer Support Bot ghi789 Response Quality ``` ## Error Handling All scripts handle common error cases: | Scenario | Output | Exit Code | | --------------- | -------------------------------------------- | --------- | | No API key set | `Error: EVALPAL_API_KEY is not set` | 1 | | Invalid API key | `Error: Authentication failed (401)` | 1 | | Eval not found | `Error: Eval definition not found (404)` | 1 | | Rate limited | `Error: Rate limited — retry after Xs (429)` | 1 | | Timeout (5 min) | `Error: Evaluation timed out after 300s` | 1 | | Network error | `Error: Could not reach EvalPal API` | 1 | ## Security - The API key is read from `EVALPAL_API_KEY` environment variable only - Scripts never echo or log the API key - All API calls use HTTPS

path: README.md
content: # EvalPal — ClawHub Skill

Run AI agent evaluations directly from your OpenClaw agent via `/evalpal` slash
commands. Trigger eval runs, check results, and list available evaluations — all
inline in Slack or Discord.

## Installation

```bash
openclaw skills install evalpal
```

## Setup

1. **Get an API key** from [evalpal.dev](https://evalpal.dev) → Settings → API
   Keys.
2. **Add the key** to your OpenClaw skill configuration:

```yaml
# In your OpenClaw agent config
skills:
  entries:
    evalpal:
      env:
        EVALPAL_API_KEY: 'sk_your_key_here'
```

## Commands

### Run an evaluation

```bash
/evalpal run --eval-id <ID>
```

Triggers a full evaluation run. The skill polls for completion (up to 5 minutes)
and prints a formatted summary with pass/fail status per test case.

**Example output:**

```
✅ Evaluation — PASSED (15/16)
├── Test Case tc_001: ✓ PASS
├── Test Case tc_002: ✓ PASS
├── Test Case tc_003: ✗ FAIL
└── 12 more passed...

Run ID: run_abc123 · 16 test cases · 47s
```

### Check run status

```bash
/evalpal status --run-id <ID>
```

Returns the current status of an in-progress or completed run.

### List evaluations

```bash
/evalpal list [--project-id <ID>]
```

Lists all evaluation definitions. Omit `--project-id` to see evals across all
projects.

## Requirements

- `curl` and `jq` must be available (standard on macOS and Linux)
- An EvalPal account with at least one evaluation definition configured

## Troubleshooting

| Problem                       | Fix                                                  |
| ----------------------------- | ---------------------------------------------------- |
| `EVALPAL_API_KEY is not set`  | Add your API key to the OpenClaw skill env config    |
| `Authentication failed (401)` | Verify your API key is valid and hasn't been revoked |
| `Eval definition not found`   | Run `/evalpal list` to find valid eval IDs           |
| `Rate limited (429)`          | Wait and retry; reduce eval frequency if persistent  |
| `Could not reach EvalPal API` | Check network connectivity and `EVALPAL_API_URL`     |

## Links

- [EvalPal](https://evalpal.dev) — AI evaluation platform
- [EvalPal Docs](https://evalpal.dev/api/docs) — API documentation
- [OpenClaw](https://openclaw.ai) — AI agent platform
- [ClawHub](https://clawhub.ai) — Skill marketplace


path: scripts/check-status.sh
content: #!/bin/sh
# check-status.sh — Check the status of an EvalPal evaluation run.
# Usage: bash scripts/check-status.sh --run-id <RUN_ID>
#
# Environment:
#   EVALPAL_API_KEY  (required) — API key for authentication
#   EVALPAL_API_URL  (optional) — Base URL (default: https://evalpal.dev)

set -e

# ---------------------------------------------------------------------------
# Defaults & argument parsing
# ---------------------------------------------------------------------------
API_URL="${EVALPAL_API_URL:-https://evalpal.dev}"
RUN_ID=""

while [ $# -gt 0 ]; do
  case "$1" in
    --run-id)  RUN_ID="$2";  shift 2 ;;
    --help|-h)
      echo "Usage: bash check-status.sh --run-id <ID>"
      echo ""
      echo "Environment:"
      echo "  EVALPAL_API_KEY   API key (required)"
      echo "  EVALPAL_API_URL   Base URL (default: https://evalpal.dev)"
      exit 0
      ;;
    *)  echo "Error: Unknown argument: $1"; exit 1 ;;
  esac
done

# ---------------------------------------------------------------------------
# Validation
# ---------------------------------------------------------------------------
if [ -z "$EVALPAL_API_KEY" ]; then
  echo "Error: EVALPAL_API_KEY is not set"
  exit 1
fi

if [ -z "$RUN_ID" ]; then
  echo "Error: --run-id is required"
  exit 1
fi

for tool in curl jq; do
  if ! command -v "$tool" >/dev/null 2>&1; then
    echo "Error: '$tool' is required but not installed"
    exit 1
  fi
done

# ---------------------------------------------------------------------------
# Fetch run status
# ---------------------------------------------------------------------------
RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
  -H "Authorization: Bearer $EVALPAL_API_KEY" \
  -H "Content-Type: application/json" \
  "${API_URL}/api/v1/runs/${RUN_ID}" 2>/dev/null) || {
  echo "Error: Could not reach EvalPal API" >&2
  exit 1
}

HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')

case "$HTTP_CODE" in
  2[0-9][0-9]) ;;
  401) echo "Error: Authentication failed (401)" >&2; exit 1 ;;
  404) echo "Error: Run not found (404)" >&2; exit 1 ;;
  429)
    RETRY=$(echo "$BODY" | jq -r '.retryAfter // "60"' 2>/dev/null || echo "60")
    echo "Error: Rate limited — retry after ${RETRY}s (429)" >&2
    exit 1
    ;;
  *)  echo "Error: API returned HTTP $HTTP_CODE" >&2; exit 1 ;;
esac

# ---------------------------------------------------------------------------
# Format output
# ---------------------------------------------------------------------------
STATUS=$(echo "$BODY" | jq -r '.status // "unknown"')
TOTAL=$(echo "$BODY" | jq -r '.totalTests // "-"')
PASSED=$(echo "$BODY" | jq -r '.passedTests // "-"')
FAILED=$(echo "$BODY" | jq -r '.failedTests // "-"')
CREATED=$(echo "$BODY" | jq -r '.createdAt // "-"')

echo "📊 Run Status: $RUN_ID"
echo "Status: $STATUS"
echo "Started: $CREATED"

if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
  echo "Total: $TOTAL · Passed: $PASSED · Failed: $FAILED"
fi


path: scripts/list-evals.sh
content: #!/bin/sh
# list-evals.sh — List available EvalPal evaluation definitions.
# Usage: bash scripts/list-evals.sh [--project-id <PROJECT_ID>]
#
# Environment:
#   EVALPAL_API_KEY  (required) — API key for authentication
#   EVALPAL_API_URL  (optional) — Base URL (default: https://evalpal.dev)

set -e

# ---------------------------------------------------------------------------
# Defaults & argument parsing
# ---------------------------------------------------------------------------
API_URL="${EVALPAL_API_URL:-https://evalpal.dev}"
PROJECT_ID=""

while [ $# -gt 0 ]; do
  case "$1" in
    --project-id)  PROJECT_ID="$2";  shift 2 ;;
    --help|-h)
      echo "Usage: bash list-evals.sh [--project-id <ID>]"
      echo ""
      echo "Lists all evaluation definitions. Optionally filter by project."
      echo ""
      echo "Environment:"
      echo "  EVALPAL_API_KEY   API key (required)"
      echo "  EVALPAL_API_URL   Base URL (default: https://evalpal.dev)"
      exit 0
      ;;
    *)  echo "Error: Unknown argument: $1"; exit 1 ;;
  esac
done

# ---------------------------------------------------------------------------
# Validation
# ---------------------------------------------------------------------------
if [ -z "$EVALPAL_API_KEY" ]; then
  echo "Error: EVALPAL_API_KEY is not set"
  exit 1
fi

for tool in curl jq; do
  if ! command -v "$tool" >/dev/null 2>&1; then
    echo "Error: '$tool' is required but not installed"
    exit 1
  fi
done

# ---------------------------------------------------------------------------
# Helper: authenticated GET request
# ---------------------------------------------------------------------------
api_get() {
  _path="$1"

  _response=$(curl -s -w "\n%{http_code}" -X GET \
    -H "Authorization: Bearer $EVALPAL_API_KEY" \
    -H "Content-Type: application/json" \
    "${API_URL}${_path}" 2>/dev/null) || {
    echo "Error: Could not reach EvalPal API" >&2
    exit 1
  }

  _http_code=$(echo "$_response" | tail -n1)
  _body=$(echo "$_response" | sed '$d')

  case "$_http_code" in
    2[0-9][0-9]) echo "$_body" ;;
    401) echo "Error: Authentication failed (401)" >&2; exit 1 ;;
    429)
      _retry=$(echo "$_body" | jq -r '.retryAfter // "60"' 2>/dev/null || echo "60")
      echo "Error: Rate limited — retry after ${_retry}s (429)" >&2
      exit 1
      ;;
    *)  echo "Error: API returned HTTP $_http_code" >&2; exit 1 ;;
  esac
}

# ---------------------------------------------------------------------------
# Fetch and display
# ---------------------------------------------------------------------------
echo "📋 Evaluation Definitions"
echo ""

if [ -n "$PROJECT_ID" ]; then
  # Single project mode
  EVALS=$(api_get "/api/v1/projects/${PROJECT_ID}/eval-definitions")
  EVAL_COUNT=$(echo "$EVALS" | jq 'length')

  if [ "$EVAL_COUNT" -eq 0 ]; then
    echo "  (no evaluations found)"
  else
    i=0
    while [ "$i" -lt "$EVAL_COUNT" ]; do
      EVAL_ID=$(echo "$EVALS" | jq -r ".[$i].id")
      EVAL_NAME=$(echo "$EVALS" | jq -r ".[$i].name")
      echo "  $EVAL_ID  $EVAL_NAME"
      i=$((i + 1))
    done
  fi
else
  # All projects mode
  PROJECTS=$(api_get "/api/v1/projects")
  PROJECT_COUNT=$(echo "$PROJECTS" | jq 'length')

  if [ "$PROJECT_COUNT" -eq 0 ]; then
    echo "  (no projects found)"
    exit 0
  fi

  p=0
  while [ "$p" -lt "$PROJECT_COUNT" ]; do
    P_ID=$(echo "$PROJECTS" | jq -r ".[$p].id")
    P_NAME=$(echo "$PROJECTS" | jq -r ".[$p].name")

    echo "Project: $P_NAME"

    EVALS=$(api_get "/api/v1/projects/${P_ID}/eval-definitions")
    EVAL_COUNT=$(echo "$EVALS" | jq 'length')

    if [ "$EVAL_COUNT" -eq 0 ]; then
      echo "  (no evaluations)"
    else
      i=0
      while [ "$i" -lt "$EVAL_COUNT" ]; do
        EVAL_ID=$(echo "$EVALS" | jq -r ".[$i].id")
        EVAL_NAME=$(echo "$EVALS" | jq -r ".[$i].name")
        echo "  $EVAL_ID  $EVAL_NAME"
        i=$((i + 1))
      done
    fi

    echo ""
    p=$((p + 1))
  done
fi


path: scripts/run-eval.sh
content: #!/bin/sh
# run-eval.sh — Trigger an EvalPal evaluation run, poll for completion, and print results.
# Usage: bash scripts/run-eval.sh --eval-id <EVAL_DEFINITION_ID>
#
# Environment:
#   EVALPAL_API_KEY  (required) — API key for authentication
#   EVALPAL_API_URL  (optional) — Base URL (default: https://evalpal.dev)

set -e

# ---------------------------------------------------------------------------
# Defaults & argument parsing
# ---------------------------------------------------------------------------
API_URL="${EVALPAL_API_URL:-https://evalpal.dev}"
EVAL_ID=""

while [ $# -gt 0 ]; do
  case "$1" in
    --eval-id)  EVAL_ID="$2";  shift 2 ;;
    --help|-h)
      echo "Usage: bash run-eval.sh --eval-id <ID>"
      echo ""
      echo "Environment:"
      echo "  EVALPAL_API_KEY   API key (required)"
      echo "  EVALPAL_API_URL   Base URL (default: https://evalpal.dev)"
      exit 0
      ;;
    *)  echo "Error: Unknown argument: $1"; exit 1 ;;
  esac
done

# ---------------------------------------------------------------------------
# Validation
# ---------------------------------------------------------------------------
if [ -z "$EVALPAL_API_KEY" ]; then
  echo "Error: EVALPAL_API_KEY is not set"
  exit 1
fi

if [ -z "$EVAL_ID" ]; then
  echo "Error: --eval-id is required"
  exit 1
fi

# Check for required tools
for tool in curl jq; do
  if ! command -v "$tool" >/dev/null 2>&1; then
    echo "Error: '$tool' is required but not installed"
    exit 1
  fi
done

# ---------------------------------------------------------------------------
# Helper: make an authenticated API request
# ---------------------------------------------------------------------------
api_request() {
  _method="$1"
  _path="$2"
  _body="${3:-}"

  _curl_args="-s -w \n%{http_code} -X $_method"
  _curl_args="$_curl_args -H \"Authorization: Bearer \$EVALPAL_API_KEY\""
  _curl_args="$_curl_args -H \"Content-Type: application/json\""

  if [ -n "$_body" ]; then
    _response=$(curl -s -w "\n%{http_code}" -X "$_method" \
      -H "Authorization: Bearer $EVALPAL_API_KEY" \
      -H "Content-Type: application/json" \
      -d "$_body" \
      "${API_URL}${_path}" 2>/dev/null) || {
      echo "Error: Could not reach EvalPal API" >&2
      exit 1
    }
  else
    _response=$(curl -s -w "\n%{http_code}" -X "$_method" \
      -H "Authorization: Bearer $EVALPAL_API_KEY" \
      -H "Content-Type: application/json" \
      "${API_URL}${_path}" 2>/dev/null) || {
      echo "Error: Could not reach EvalPal API" >&2
      exit 1
    }
  fi

  _http_code=$(echo "$_response" | tail -n1)
  _body_out=$(echo "$_response" | sed '$d')

  case "$_http_code" in
    2[0-9][0-9]) echo "$_body_out" ;;
    401) echo "Error: Authentication failed (401)" >&2; exit 1 ;;
    404) echo "Error: Eval definition not found (404)" >&2; exit 1 ;;
    429)
      _retry=$(echo "$_body_out" | jq -r '.retryAfter // "60"' 2>/dev/null || echo "60")
      echo "Error: Rate limited — retry after ${_retry}s (429)" >&2
      exit 1
      ;;
    *)  echo "Error: API returned HTTP $_http_code" >&2; exit 1 ;;
  esac
}

# ---------------------------------------------------------------------------
# Step 1: Trigger the eval run
# ---------------------------------------------------------------------------
echo "Starting evaluation run for eval definition: $EVAL_ID"

RUN_RESPONSE=$(api_request POST "/api/v1/evals/${EVAL_ID}/run" "{}")
RUN_ID=$(echo "$RUN_RESPONSE" | jq -r '.id')

if [ -z "$RUN_ID" ] || [ "$RUN_ID" = "null" ]; then
  echo "Error: Failed to create eval run"
  exit 1
fi

echo "Run created: $RUN_ID"

# ---------------------------------------------------------------------------
# Step 2: Poll for completion (exponential backoff, 5-minute timeout)
# ---------------------------------------------------------------------------
DELAY=2
MAX_DELAY=10
ELAPSED=0
TIMEOUT=300

while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
  STATUS_RESPONSE=$(api_request GET "/api/v1/runs/${RUN_ID}")
  STATUS=$(echo "$STATUS_RESPONSE" | jq -r '.status')

  case "$STATUS" in
    completed|failed) break ;;
    *)
      sleep "$DELAY"
      ELAPSED=$((ELAPSED + DELAY))
      DELAY=$((DELAY + DELAY / 2))
      [ "$DELAY" -gt "$MAX_DELAY" ] && DELAY=$MAX_DELAY
      ;;
  esac
done

if [ "$ELAPSED" -ge "$TIMEOUT" ]; then
  echo "Error: Evaluation timed out after ${TIMEOUT}s"
  exit 1
fi

# ---------------------------------------------------------------------------
# Step 3: Fetch results
# ---------------------------------------------------------------------------
RESULTS_RESPONSE=$(api_request GET "/api/v1/runs/${RUN_ID}/results")

# ---------------------------------------------------------------------------
# Step 4: Format output
# ---------------------------------------------------------------------------
TOTAL=$(echo "$STATUS_RESPONSE" | jq -r '.totalTests // 0')
PASSED=$(echo "$STATUS_RESPONSE" | jq -r '.passedTests // 0')
FAILED=$(echo "$STATUS_RESPONSE" | jq -r '.failedTests // 0')

if [ "$STATUS" = "completed" ] && [ "$FAILED" -eq 0 ]; then
  echo ""
  echo "✅ Evaluation — PASSED ($PASSED/$TOTAL)"
else
  echo ""
  echo "❌ Evaluation — FAILED ($PASSED/$TOTAL)"
fi

# Print per-test-case results
RESULT_COUNT=$(echo "$RESULTS_RESPONSE" | jq 'length')
if [ "$RESULT_COUNT" -gt 0 ]; then
  i=0
  while [ "$i" -lt "$RESULT_COUNT" ]; do
    TC_ID=$(echo "$RESULTS_RESPONSE" | jq -r ".[$i].testCaseId")
    TC_PASS=$(echo "$RESULTS_RESPONSE" | jq -r ".[$i].overallPass")

    if [ "$i" -eq $((RESULT_COUNT - 1)) ]; then
      PREFIX="└──"
    else
      PREFIX="├──"
    fi

    if [ "$TC_PASS" = "true" ]; then
      echo "$PREFIX Test Case $TC_ID: ✓ PASS"
    else
      echo "$PREFIX Test Case $TC_ID: ✗ FAIL"
    fi

    i=$((i + 1))
  done
fi

# Summary line
echo ""
echo "Run ID: $RUN_ID · $TOTAL test cases · ${ELAPSED}s"

# Exit code
if [ "$STATUS" = "failed" ] || [ "$FAILED" -gt 0 ]; then
  exit 1
fi

exit 0


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)