Skill: akellacom/prometheus (version 1.1.0)

--- name: prometheus description: Query Prometheus monitoring data to check server metrics, resource usage, and system health. Use when the user asks about server status, disk space, CPU/memory usage, network stats, or any metrics collected by Prometheus. Supports multiple Prometheus instances with aggregated queries, config file or environment variables, and HTTP Basic Auth. --- # Prometheus Skill Query Prometheus monitoring data from one or multiple instances. Supports federation across multiple Prometheus servers with a single command. ## Quick Start ### 1. Initial Setup Run the interactive configuration wizard: ```bash cd ~/.openclaw/workspace/skills/prometheus node scripts/cli.js init ``` This will create a `prometheus.json` config file in your OpenClaw workspace (`~/.openclaw/workspace/prometheus.json`). ### 2. Start Querying ```bash # Query default instance node scripts/cli.js query 'up' # Query all instances at once node scripts/cli.js query 'up' --all # List configured instances node scripts/cli.js instances ``` ## Configuration ### Config File Location By default, the skill looks for config in your OpenClaw workspace: ``` ~/.openclaw/workspace/prometheus.json ``` **Priority order:** 1. Path from `PROMETHEUS_CONFIG` environment variable 2. `~/.openclaw/workspace/prometheus.json` 3. `~/.openclaw/workspace/config/prometheus.json` 4. `./prometheus.json` (current directory) 5. `~/.config/prometheus/config.json` ### Config Format Create `prometheus.json` in your workspace (or use `node cli.js init`): ```json { "instances": [ { "name": "production", "url": "https://prometheus.example.com", "user": "admin", "password": "secret" }, { "name": "staging", "url": "http://prometheus-staging:9090" } ], "default": "production" } ``` **Fields:** - `name` — unique identifier for the instance - `url` — Prometheus server URL - `user` / `password` — optional HTTP Basic Auth credentials - `default` — which instance to use when none specified ### Environment Variables (Legacy) For single-instance setups, you can use environment variables: ```bash export PROMETHEUS_URL=https://prometheus.example.com export PROMETHEUS_USER=admin # optional export PROMETHEUS_PASSWORD=secret # optional ``` ## Usage ### Global Flags | Flag | Description | |------|-------------| | `-c, --config <path>` | Path to config file | | `-i, --instance <name>` | Target specific instance | | `-a, --all` | Query all configured instances | ### Commands #### Setup ```bash # Interactive configuration wizard node scripts/cli.js init ``` #### Query Metrics ```bash cd ~/.openclaw/workspace/skills/prometheus # Query default instance node scripts/cli.js query 'up' # Query specific instance node scripts/cli.js query 'up' -i staging # Query ALL instances at once node scripts/cli.js query 'up' --all # Custom config file node scripts/cli.js query 'up' -c /path/to/config.json ``` #### Common Queries **Disk space usage:** ```bash node scripts/cli.js query '100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes * 100)' --all ``` **CPU usage:** ```bash node scripts/cli.js query '100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)' --all ``` **Memory usage:** ```bash node scripts/cli.js query '(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100' --all ``` **Load average:** ```bash node scripts/cli.js query 'node_load1' --all ``` ### List Configured Instances ```bash node scripts/cli.js instances ``` Output: ```json { "default": "production", "instances": [ { "name": "production", "url": "https://prometheus.example.com", "hasAuth": true }, { "name": "staging", "url": "http://prometheus-staging:9090", "hasAuth": false } ] } ``` ### Other Commands ```bash # List all metrics matching pattern node scripts/cli.js metrics 'node_memory_*' # Get label names node scripts/cli.js labels --all # Get values for a label node scripts/cli.js label-values instance --all # Find time series node scripts/cli.js series '{__name__=~"node_cpu_.*", instance=~".*:9100"}' --all # Get active alerts node scripts/cli.js alerts --all # Get scrape targets node scripts/cli.js targets --all ``` ## Multi-Instance Output Format When using `--all`, results include data from all instances: ```json { "resultType": "vector", "results": [ { "instance": "production", "status": "success", "resultType": "vector", "result": [...] }, { "instance": "staging", "status": "success", "resultType": "vector", "result": [...] } ] } ``` Errors on individual instances don't fail the entire query — they appear with `"status": "error"` in the results array. ## Common Queries Reference | Metric | PromQL Query | |--------|--------------| | Disk free % | `node_filesystem_avail_bytes / node_filesystem_size_bytes * 100` | | Disk used % | `100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes * 100)` | | CPU idle % | `avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100` | | Memory used % | `(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100` | | Network RX | `rate(node_network_receive_bytes_total[5m])` | | Network TX | `rate(node_network_transmit_bytes_total[5m])` | | Uptime | `node_time_seconds - node_boot_time_seconds` | | Service up | `up` | ## Notes - Time range defaults to last 1 hour for instant queries - Use range queries `[5m]` for rate calculations - All queries return JSON with `data.result` containing the results - Instance labels typically show `host:port` format - When using `--all`, queries run in parallel for faster results - Config is stored outside the skill directory so it persists across skill updates

path: package.json
content: {
  "name": "prometheus-cli",
  "version": "1.1.0",
  "description": "CLI for querying Prometheus monitoring data with multi-instance support",
  "type": "module",
  "engines": {
    "node": ">=18.0.0"
  },
  "scripts": {
    "query": "node scripts/cli.js query",
    "labels": "node scripts/cli.js labels",
    "alerts": "node scripts/cli.js alerts",
    "targets": "node scripts/cli.js targets"
  }
}


path: scripts/cli.js
content: #!/usr/bin/env node

/**
 * Prometheus CLI - Query multiple Prometheus instances
 * 
 * Usage:
 *   node cli.js init                              - Interactive config setup
 *   node cli.js query '<promql>'                  - Run instant query
 *   node cli.js query '<promql>' --all            - Run query on all instances
 *   node cli.js query '<promql>' -i prod          - Run query on specific instance
 *   node cli.js range '<promql>' <start> <end>    - Run range query
 *   node cli.js instances                         - List configured instances
 * 
 * Global flags:
 *   -c, --config <path>     Path to config file
 *   -i, --instance <name>   Target specific instance
 *   -a, --all              Query all instances
 */

import { 
  instantQuery, 
  rangeQuery, 
  getLabels, 
  getLabelValues, 
  getSeries, 
  getMetadata,
  getAlerts,
  getTargets
} from './query.js';
import { loadConfig, getAllInstances } from './common.js';
import { existsSync, writeFileSync, mkdirSync } from 'fs';
import { createInterface } from 'readline';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';

const __dirname = dirname(fileURLToPath(import.meta.url));

/**
 * Get OpenClaw workspace directory
 */
function getWorkspaceDir() {
  let dir = __dirname;
  while (dir !== '/') {
    if (existsSync(join(dir, 'AGENTS.md')) || existsSync(join(dir, 'SOUL.md'))) {
      return dir;
    }
    dir = dirname(dir);
  }
  return process.env.OPENCLAW_WORKSPACE || join(process.env.HOME || '', '.openclaw', 'workspace');
}

// Parse arguments
const args = process.argv.slice(2);
const command = args[0];

// Extract global flags
let configPath = null;
let targetInstance = null;
let queryAll = false;
const remainingArgs = [];

for (let i = 0; i < args.length; i++) {
  const arg = args[i];
  if (arg === '-c' || arg === '--config') {
    configPath = args[++i];
  } else if (arg === '-i' || arg === '--instance') {
    targetInstance = args[++i];
  } else if (arg === '-a' || arg === '--all') {
    queryAll = true;
  } else {
    remainingArgs.push(arg);
  }
}

// Set config path if provided
if (configPath) {
  process.env.PROMETHEUS_CONFIG = configPath;
}

function output(data) {
  console.log(JSON.stringify(data, null, 2));
}

function errorExit(message) {
  console.error(`Error: ${message}`);
  process.exit(1);
}

async function prompt(rl, question, defaultValue = '') {
  return new Promise((resolve) => {
    const text = defaultValue 
      ? `${question} (${defaultValue}): `
      : `${question}: `;
    rl.question(text, (answer) => {
      resolve(answer.trim() || defaultValue);
    });
  });
}

async function initConfig() {
  const workspaceDir = getWorkspaceDir();
  const defaultConfigPath = join(workspaceDir, 'prometheus.json');
  const configFile = process.env.PROMETHEUS_CONFIG || defaultConfigPath;
  
  console.log('\n🎯 Prometheus CLI Configuration\n');
  console.log(`Config will be saved to: ${configFile}\n`);
  
  const rl = createInterface({
    input: process.stdin,
    output: process.stdout
  });
  
  const instances = [];
  let addMore = true;
  
  while (addMore) {
    console.log(`\n📡 Instance #${instances.length + 1}`);
    
    const name = await prompt(rl, 'Name (e.g., prod, staging)', instances.length === 0 ? 'main' : `instance-${instances.length + 1}`);
    const url = await prompt(rl, 'URL (e.g., http://localhost:9090)');
    
    if (!url) {
      console.log('❌ URL is required, skipping this instance');
      break;
    }
    
    const auth = await prompt(rl, 'Use HTTP Basic Auth? (y/N)');
    let user = null;
    let password = null;
    
    if (auth.toLowerCase() === 'y') {
      user = await prompt(rl, 'Username');
      password = await prompt(rl, 'Password');
    }
    
    instances.push({
      name,
      url: url.replace(/\/$/, ''), // Remove trailing slash
      ...(user && { user }),
      ...(password && { password })
    });
    
    const more = await prompt(rl, 'Add another instance? (y/N)');
    addMore = more.toLowerCase() === 'y';
  }
  
  if (instances.length === 0) {
    console.log('\n❌ No instances configured. Aborting.');
    rl.close();
    process.exit(1);
  }
  
  let defaultInstance = instances[0].name;
  if (instances.length > 1) {
    const names = instances.map(i => i.name).join(', ');
    defaultInstance = await prompt(rl, `Default instance (${names})`, instances[0].name);
  }
  
  rl.close();
  
  const config = {
    instances,
    default: defaultInstance
  };
  
  try {
    // Ensure directory exists
    const configDir = dirname(configFile);
    if (!existsSync(configDir)) {
      mkdirSync(configDir, { recursive: true });
    }
    
    writeFileSync(configFile, JSON.stringify(config, null, 2));
    console.log(`\n✅ Configuration saved to ${configFile}`);
    console.log(`\nConfigured instances:`);
    instances.forEach(i => {
      const auth = i.user ? ' (with auth)' : '';
      const def = i.name === defaultInstance ? ' [default]' : '';
      console.log(`  • ${i.name}: ${i.url}${auth}${def}`);
    });
    console.log('\nYou can now run queries:\n');
    console.log('  node cli.js query \'up\'');
    console.log('  node cli.js query \'node_load1\' --all');
    console.log(`  node cli.js query \'up\' -i ${instances[0].name}`);
  } catch (err) {
    console.error(`\n❌ Failed to save config: ${err.message}`);
    process.exit(1);
  }
}

async function main() {
  try {
    switch (command) {
      case 'init': {
        await initConfig();
        break;
      }

      case 'query':
      case 'q': {
        const queryStr = remainingArgs[1];
        if (!queryStr) errorExit('Query string required');
        
        const result = await instantQuery(queryStr, {
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'range':
      case 'r': {
        const queryStr = remainingArgs[1];
        const start = remainingArgs[2];
        const end = remainingArgs[3];
        const step = parseInt(remainingArgs[4]) || 60;
        
        if (!queryStr || !start || !end) {
          errorExit('Usage: range <query> <start> <end> [step]');
        }
        
        const result = await rangeQuery(queryStr, start, end, {
          instance: targetInstance,
          all: queryAll,
          step
        });
        output(result);
        break;
      }

      case 'labels':
      case 'l': {
        const result = await getLabels({
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'label-values':
      case 'lv': {
        const label = remainingArgs[1];
        if (!label) errorExit('Label name required');
        
        const result = await getLabelValues(label, {
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'series':
      case 's': {
        const match = remainingArgs[1];
        if (!match) errorExit('Series selector required (e.g., \'{__name__="up"}\')');
        
        const result = await getSeries(match, {
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'metrics':
      case 'm': {
        const pattern = remainingArgs[1] || '';
        const result = await getMetadata(pattern, {
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'alerts':
      case 'a': {
        const result = await getAlerts({
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'targets':
      case 't': {
        const result = await getTargets({
          instance: targetInstance,
          all: queryAll
        });
        output(result);
        break;
      }

      case 'instances':
      case 'list': {
        const config = loadConfig();
        const instances = getAllInstances();
        output({
          default: config.default,
          instances: instances.map(i => ({
            name: i.name,
            url: i.url,
            hasAuth: !!(i.user && i.password)
          }))
        });
        break;
      }

      case 'help':
      case 'h':
      case '--help':
      case '-h':
      default: {
        const workspaceDir = getWorkspaceDir();
        const defaultConfigPath = join(workspaceDir, 'prometheus.json');
        
        console.log(`
Prometheus CLI - Query multiple Prometheus instances

Setup:
  init                     Interactive configuration wizard

Commands:
  query <promql>           Run instant query
  range <q> <start> <end>  Run range query (timestamps in RFC3339 or Unix)
  labels                   List all label names
  label-values <label>     Get values for a specific label
  series <selector>        Find time series by label matchers
  metrics [pattern]        Get metrics metadata
  alerts                   Get active alerts
  targets                  Get scrape targets
  instances                List configured instances

Global flags:
  -c, --config <path>      Path to config file
  -i, --instance <name>    Target specific instance
  -a, --all                Query all configured instances

Examples:
  # Initial setup
  node cli.js init
  
  # Query default instance
  node cli.js query 'up'
  
  # Query specific instance
  node cli.js query 'up' -i prod
  
  # Query all instances
  node cli.js query 'up' --all

Config file locations (searched in order):
  1. Path from PROMETHEUS_CONFIG env var
  2. ${defaultConfigPath}
  3. ~/.config/prometheus/config.json

Config file format (prometheus.json):
  {
    "instances": [
      { "name": "prod", "url": "http://prom-prod:9090" },
      { "name": "staging", "url": "http://prom-staging:9090", "user": "admin", "password": "secret" }
    ],
    "default": "prod"
  }
`);
        break;
      }
    }
  } catch (err) {
    const workspaceDir = getWorkspaceDir();
    const defaultConfigPath = join(workspaceDir, 'prometheus.json');
    
    if (err.message.includes('No Prometheus configuration found') || 
        err.message.includes('No instances configured')) {
      console.error(`
❌ Configuration not found!

To get started, run the setup wizard:
  node cli.js init

Or create a config file manually at:
  ${defaultConfigPath}

Example config:
  {
    "instances": [
      { "name": "main", "url": "http://localhost:9090" }
    ],
    "default": "main"
  }
`);
      process.exit(1);
    }
    errorExit(err.message);
  }
}

main();


path: scripts/common.js
content: /**
 * Common utilities for Prometheus API client
 * Supports multiple instances via config file
 */

import { readFileSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';

const __dirname = dirname(fileURLToPath(import.meta.url));
let configCache = null;
let envLoaded = false;

/**
 * Get OpenClaw workspace directory
 * @returns {string} Path to workspace
 */
function getWorkspaceDir() {
  // Try to find workspace by walking up from current file
  let dir = __dirname;
  while (dir !== '/') {
    if (existsSync(join(dir, 'AGENTS.md')) || existsSync(join(dir, 'SOUL.md'))) {
      return dir;
    }
    dir = dirname(dir);
  }
  // Fallback to standard location
  return process.env.OPENCLAW_WORKSPACE || join(process.env.HOME || '', '.openclaw', 'workspace');
}

/**
 * Load environment variables from .env file
 */
function loadEnvFile() {
  if (envLoaded) return;
  
  const workspaceDir = getWorkspaceDir();
  const envPaths = [
    join(workspaceDir, '.env'),
    join(process.cwd(), '.env'),
  ];
  
  for (const envPath of envPaths) {
    if (existsSync(envPath)) {
      try {
        const content = readFileSync(envPath, 'utf8');
        const lines = content.split('\n');
        
        for (const line of lines) {
          const trimmed = line.trim();
          // Skip comments and empty lines
          if (!trimmed || trimmed.startsWith('#')) continue;
          
          const match = trimmed.match(/^([A-Za-z_][A-Za-z0-9_]*)=(.*)$/);
          if (match) {
            const [, key, value] = match;
            // Only set if not already in env
            if (!process.env[key]) {
              // Remove quotes if present
              const cleanValue = value.replace(/^["']|["']$/g, '');
              process.env[key] = cleanValue;
            }
          }
        }
      } catch (err) {
        // Silently ignore .env read errors
      }
    }
  }
  
  envLoaded = true;
}

/**
 * Load config from file or environment
 * @returns {Object} Config object with instances array
 */
export function loadConfig() {
  if (configCache) return configCache;

  // Load .env file first for fallback
  loadEnvFile();

  const workspaceDir = getWorkspaceDir();

  // Try to load from config file (priority: workspace -> env -> local -> home)
  const configPaths = [
    process.env.PROMETHEUS_CONFIG,
    join(workspaceDir, 'prometheus.json'),
    join(workspaceDir, 'config', 'prometheus.json'),
    './prometheus.json',
    './config.json',
    join(process.env.HOME || '', '.config', 'prometheus', 'config.json'),
  ].filter(Boolean);

  for (const configPath of configPaths) {
    if (existsSync(configPath)) {
      try {
        const content = readFileSync(configPath, 'utf8');
        const parsed = JSON.parse(content);
        
        // Validate config structure
        if (!parsed.instances || !Array.isArray(parsed.instances)) {
          throw new Error(`Invalid config: "instances" array is required`);
        }
        
        if (parsed.instances.length === 0) {
          throw new Error(`No instances configured. Run: node cli.js init`);
        }
        
        configCache = parsed;
        return configCache;
      } catch (err) {
        if (err.message.includes('Run:')) throw err;
        console.error(`Warning: Failed to load config from ${configPath}: ${err.message}`);
      }
    }
  }

  // Fallback to environment variables (backward compatibility)
  const url = process.env.PROMETHEUS_URL;
  if (url) {
    configCache = {
      instances: [{
        name: 'default',
        url: url,
        user: process.env.PROMETHEUS_USER || null,
        [REDACTED_SECRET] || null
      }],
      default: 'default'
    };
    return configCache;
  }

  throw new Error('No Prometheus configuration found. Run: node cli.js init');
}

/**
 * Get all configured instances
 * @returns {Array} Array of instance objects
 */
export function getAllInstances() {
  const config = loadConfig();
  return config.instances || [];
}

/**
 * Get a specific instance by name
 * @param {string} name - Instance name
 * @returns {Object} Instance configuration
 */
export function getInstance(name = null) {
  const config = loadConfig();
  const instances = config.instances || [];
  
  // If no name specified, use default
  const targetName = name || config.default;
  if (!targetName) {
    throw new Error('No instance specified and no default configured');
  }

  const instance = instances.find(i => i.name === targetName);
  if (!instance) {
    const available = instances.map(i => i.name).join(', ');
    throw new Error(`Instance "${targetName}" not found. Available: ${available}`);
  }

  return instance;
}

/**
 * Create authentication headers for a specific instance
 * @param {Object} instance - Instance configuration
 * @returns {Object} Headers object
 */
export function createAuthHeader(instance = null) {
  const headers = {
    'Accept': 'application/json'
  };
  
  const user = instance?.user || process.env.PROMETHEUS_USER;
  const [REDACTED_SECRET] || process.env.PROMETHEUS_PASSWORD;
  
  if (user && password) {
    const auth = Buffer.from(`${user}:${password}`).toString('base64');
    headers['Authorization'] = `Basic ${auth}`;
  }
  
  return headers;
}

/**
 * Build full URL for API endpoint
 * @param {Object} instance - Instance configuration
 * @param {string} path - API path
 * @returns {string} Full URL
 */
export function buildUrl(instance, path) {
  const baseUrl = instance.url.replace(/\/$/, ''); // Remove trailing slash
  return `${baseUrl}${path}`;
}

/**
 * Handle API response and parse JSON
 * @param {Response} response - Fetch response
 * @returns {Promise<any>} Parsed response data
 */
export async function handleResponse(response) {
  if (!response.ok) {
    const text = await response.text().catch(() => 'Unknown error');
    throw new Error(`HTTP ${response.status}: ${text}`);
  }
  
  const data = await response.json();
  
  if (data.status !== 'success') {
    throw new Error(`Prometheus error: ${data.error || 'Unknown error'}`);
  }
  
  return data.data;
}

/**
 * Format bytes to human readable string
 * @param {number} bytes - Bytes to format
 * @returns {string} Formatted string
 */
export function formatBytes(bytes) {
  if (bytes === 0) return '0 B';
  const k = 1024;
  const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
  const i = Math.floor(Math.log(bytes) / Math.log(k));
  return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}

/**
 * Format seconds to human readable duration
 * @param {number} seconds - Seconds to format
 * @returns {string} Formatted string
 */
export function formatDuration(seconds) {
  const days = Math.floor(seconds / 86400);
  const hours = Math.floor((seconds % 86400) / 3600);
  const mins = Math.floor((seconds % 3600) / 60);
  
  if (days > 0) return `${days}d ${hours}h`;
  if (hours > 0) return `${hours}h ${mins}m`;
  return `${mins}m`;
}


path: scripts/query.js
content: import { getAllInstances, getInstance, createAuthHeader, buildUrl, handleResponse } from './common.js';

/**
 * Execute instant query against one or more Prometheus instances
 * @param {string} query - PromQL query string
 * @param {Object} options - Query options
 * @param {string} options.instance - Instance name (null for default)
 * @param {boolean} options.all - Query all instances
 * @param {string} options.time - Optional Unix timestamp
 * @returns {Promise<Object>} Query result(s)
 */
export async function instantQuery(query, options = {}) {
  const { instance, all = false, time = null } = options;

  if (all) {
    return queryAllInstances('instant', query, { time });
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/query');
  const params = new URLSearchParams({ query });
  if (time) params.append('time', time);
  
  const response = await fetch(`${url}?${params}`, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    ...result
  };
}

/**
 * Execute range query against one or more Prometheus instances
 * @param {string} query - PromQL query string
 * @param {string} start - Start time (RFC3339 or Unix timestamp)
 * @param {string} end - End time (RFC3339 or Unix timestamp)
 * @param {Object} options - Query options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @param {number} options.step - Query resolution step width in seconds
 * @returns {Promise<Object>} Query result(s)
 */
export async function rangeQuery(query, start, end, options = {}) {
  const { instance, all = false, step = 60 } = options;

  if (all) {
    return queryAllInstances('range', query, { start, end, step });
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/query_range');
  const params = new URLSearchParams({ query, start, end, step: String(step) });
  
  const response = await fetch(`${url}?${params}`, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    ...result
  };
}

/**
 * Query all configured instances and aggregate results
 * @param {string} type - Query type ('instant' or 'range')
 * @param {string} query - PromQL query
 * @param {Object} params - Query parameters
 * @returns {Promise<Object>} Aggregated results
 */
async function queryAllInstances(type, query, params) {
  const instances = getAllInstances();
  
  const results = await Promise.allSettled(
    instances.map(async (inst) => {
      try {
        let result;
        if (type === 'instant') {
          const url = buildUrl(inst, '/api/v1/query');
          const searchParams = new URLSearchParams({ query });
          if (params.time) searchParams.append('time', params.time);
          
          const response = await fetch(`${url}?${searchParams}`, {
            headers: createAuthHeader(inst)
          });
          result = await handleResponse(response);
        } else {
          const url = buildUrl(inst, '/api/v1/query_range');
          const searchParams = new URLSearchParams({
            query,
            start: params.start,
            end: params.end,
            step: String(params.step)
          });
          
          const response = await fetch(`${url}?${searchParams}`, {
            headers: createAuthHeader(inst)
          });
          result = await handleResponse(response);
        }
        
        return {
          instance: inst.name,
          status: 'success',
          ...result
        };
      } catch (err) {
        return {
          instance: inst.name,
          status: 'error',
          error: err.message
        };
      }
    })
  );

  return {
    resultType: type === 'instant' ? 'vector' : 'matrix',
    results: results.map(r => r.status === 'fulfilled' ? r.value : r.reason)
  };
}

/**
 * Get label names from one or all instances
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @returns {Promise<Object>} Label names
 */
export async function getLabels(options = {}) {
  const { instance, all = false } = options;

  if (all) {
    return queryAllInstancesSimple('/api/v1/labels', instance);
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/labels');
  
  const response = await fetch(url, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Get values for a specific label
 * @param {string} label - Label name
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @returns {Promise<Object>} Label values
 */
export async function getLabelValues(label, options = {}) {
  const { instance, all = false } = options;

  if (all) {
    return queryAllInstancesSimple(`/api/v1/label/${encodeURIComponent(label)}/values`, instance);
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, `/api/v1/label/${encodeURIComponent(label)}/values`);
  
  const response = await fetch(url, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Find time series by label matchers
 * @param {string} match - Series selector (e.g., '{__name__="up"}')
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @param {string} options.start - Optional start time
 * @param {string} options.end - Optional end time
 * @returns {Promise<Object>} Array of series
 */
export async function getSeries(match, options = {}) {
  const { instance, all = false, start = null, end = null } = options;

  if (all) {
    return queryAllInstancesSeries(match, { start, end });
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/series');
  const params = new URLSearchParams();
  params.append('match[]', match);
  if (start) params.append('start', start);
  if (end) params.append('end', end);
  
  const response = await fetch(`${url}?${params}`, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Get metadata about metrics
 * @param {string} metric - Optional metric name to filter
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @returns {Promise<Object>} Metrics metadata
 */
export async function getMetadata(metric = '', options = {}) {
  const { instance, all = false } = options;

  if (all) {
    return queryAllInstancesSimple('/api/v1/metadata' + (metric ? `?metric=${encodeURIComponent(metric)}` : ''), instance);
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/metadata');
  const params = metric ? `?metric=${encodeURIComponent(metric)}` : '';
  
  const response = await fetch(`${url}${params}`, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Get current alerts
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @returns {Promise<Object>} Active alerts
 */
export async function getAlerts(options = {}) {
  const { instance, all = false } = options;

  if (all) {
    return queryAllInstancesSimple('/api/v1/alerts', instance);
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/alerts');
  
  const response = await fetch(url, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Get target discovery status
 * @param {Object} options - Options
 * @param {string} options.instance - Instance name
 * @param {boolean} options.all - Query all instances
 * @returns {Promise<Object>} Scrape targets
 */
export async function getTargets(options = {}) {
  const { instance, all = false } = options;

  if (all) {
    return queryAllInstancesSimple('/api/v1/targets', instance);
  }

  const targetInstance = getInstance(instance);
  const url = buildUrl(targetInstance, '/api/v1/targets');
  
  const response = await fetch(url, {
    headers: createAuthHeader(targetInstance)
  });
  
  const result = await handleResponse(response);
  return {
    instance: targetInstance.name,
    data: result
  };
}

/**
 * Helper to query simple endpoints on all instances
 */
async function queryAllInstancesSimple(path, instance) {
  const instances = getAllInstances();
  
  const results = await Promise.allSettled(
    instances.map(async (inst) => {
      try {
        const url = buildUrl(inst, path);
        const response = await fetch(url, {
          headers: createAuthHeader(inst)
        });
        const result = await handleResponse(response);
        
        return {
          instance: inst.name,
          status: 'success',
          data: result
        };
      } catch (err) {
        return {
          instance: inst.name,
          status: 'error',
          error: err.message
        };
      }
    })
  );

  return {
    results: results.map(r => r.status === 'fulfilled' ? r.value : r.reason)
  };
}

/**
 * Helper to query series on all instances
 */
async function queryAllInstancesSeries(match, params) {
  const instances = getAllInstances();
  
  const results = await Promise.allSettled(
    instances.map(async (inst) => {
      try {
        const url = buildUrl(inst, '/api/v1/series');
        const searchParams = new URLSearchParams();
        searchParams.append('match[]', match);
        if (params.start) searchParams.append('start', params.start);
        if (params.end) searchParams.append('end', params.end);
        
        const response = await fetch(`${url}?${searchParams}`, {
          headers: createAuthHeader(inst)
        });
        const result = await handleResponse(response);
        
        return {
          instance: inst.name,
          status: 'success',
          data: result
        };
      } catch (err) {
        return {
          instance: inst.name,
          status: 'error',
          error: err.message
        };
      }
    })
  );

  return {
    results: results.map(r => r.status === 'fulfilled' ? r.value : r.reason)
  };
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)