Skill: wenbingyu/hologres-slow-query-analysis (version 0.2.0)

--- name: hologres-slow-query-analysis description: | Hologres slow query log analysis and diagnosis skill. Use for analyzing slow queries, failed queries, query performance diagnosis, and log management in Alibaba Cloud Hologres. Triggers: "hologres slow query", "hg_query_log", "query diagnosis", "慢Query分析", "Hologres性能诊断" --- ## Prerequisites This skill requires **hologres-cli** to be installed first: ```bash pip install hologres-cli export HOLOGRES_SKILL=hologres-slow-query-analysis ``` All SQL execution and GUC parameter operations depend on `hologres-cli` commands (`hologres sql run`, `hologres guc set`). # Hologres Slow Query Analysis Diagnose and analyze slow/failed queries in Alibaba Cloud Hologres using the `hologres.hg_query_log` system table. ## Version Requirements | Hologres Version | Feature | |-----------------|---------| | V0.10+ | Basic slow query log | | V2.2+ | SQL fingerprint (digest) | | V2.2.7+ | Default threshold 100ms | | V3.0.2+ | Aggregated records for <100ms queries | ## Quick Start ### 1. Check Permissions ```sql -- Superuser: view all DB logs ALTER USER "cloud_account_id" SUPERUSER; -- Or join pg_read_all_stats group GRANT pg_read_all_stats TO "cloud_account_id"; -- For current DB only (SPM model) CALL spm_grant('<db_name>_admin', 'cloud_account_id'); ``` ### 2. Basic Query Count ```sql SELECT count(*) FROM hologres.hg_query_log; ``` ### 3. Recent Slow Queries (10 min) ```sql SELECT status AS "Status", duration AS "Duration(ms)", query_start AS "Start Time", (read_bytes/1048576)::text || ' MB' AS "Read", (memory_bytes/1048576)::text || ' MB' AS "Memory", (cpu_time_ms/1000)::text || ' s' AS "CPU", query_id AS "QueryID", query::char(50) AS "Query" FROM hologres.hg_query_log WHERE query_start >= now() - interval '10 min' ORDER BY duration DESC LIMIT 100; ``` ## Core Diagnostic Workflows ### Workflow 1: Find Resource-Heavy Queries Use when CPU/memory usage is high. ```sql -- Top 10 CPU-consuming queries (past day) SELECT digest, avg(cpu_time_ms), sum(cpu_time_ms) FROM hologres.hg_query_log WHERE query_start >= CURRENT_DATE - INTERVAL '1 day' AND digest IS NOT NULL AND usename != 'system' GROUP BY 1 ORDER BY 3 DESC LIMIT 10; ``` ### Workflow 2: Find Failed Queries ```sql SELECT status, message::char(100), duration, query_start, query_id, query::char(80) FROM hologres.hg_query_log WHERE query_start BETWEEN '2024-01-01 00:00:00'::timestamptz AND '2024-01-01 01:00:00'::timestamptz AND status = 'FAILED' ORDER BY query_start ASC LIMIT 100; ``` ### Workflow 3: Query Phase Analysis Identify bottleneck phase (optimization/startup/execution). ```sql SELECT status, duration AS "Total(ms)", optimization_cost AS "Optimize(ms)", start_query_cost AS "Startup(ms)", get_next_cost AS "Execute(ms)", duration - optimization_cost - start_query_cost - get_next_cost AS "Other(ms)", query_id, query::char(50) FROM hologres.hg_query_log WHERE query_start >= now() - interval '10 min' ORDER BY duration DESC LIMIT 100; ``` ### Workflow 4: Compare with Yesterday ```sql SELECT query_date, count(1), sum(read_bytes), sum(cpu_time_ms) FROM hologres.hg_query_log WHERE query_start >= now() - interval '3 h' GROUP BY query_date UNION ALL SELECT query_date, count(1), sum(read_bytes), sum(cpu_time_ms) FROM hologres.hg_query_log WHERE query_start >= now() - interval '1d 3h' AND query_start <= now() - interval '1d' GROUP BY query_date; ``` ## Key Fields Reference | Field | Description | |-------|-------------| | `query_id` | Unique query identifier | | `digest` | SQL fingerprint (MD5 hash) | | `duration` | Total query time (ms) | | `cpu_time_ms` | CPU time consumed | | `memory_bytes` | Peak memory usage | | `read_bytes` | Data read volume | | `engine_type` | Query engine (HQE/PQE/SDK/PG) | | `optimization_cost` | Plan generation time | | `start_query_cost` | Query startup time | | `get_next_cost` | Execution time | ## Configuration ```sql -- Set slow query threshold (DB level, superuser only) ALTER DATABASE dbname SET log_min_duration_statement = '250ms'; -- Session level SET log_min_duration_statement = '250ms'; -- Set log retention (V3.0.27+, 3-30 days) ALTER DATABASE dbname SET hg_query_log_retention_time_sec = 2592000; ``` Or use the CLI for database-level settings: ```bash hologres guc set log_min_duration_statement '250ms' hologres guc set hg_query_log_retention_time_sec 2592000 ``` ## References | Document | Content | |----------|---------| | [diagnostic-queries.md](references/diagnostic-queries.md) | Complete diagnostic SQL collection | | [log-export.md](references/log-export.md) | Export logs to internal/external tables | | [configuration.md](references/configuration.md) | Configuration parameters | ## Best Practices 1. Always filter by `query_start` for better performance 2. Use `digest` to group similar queries for pattern analysis 3. Check `engine_type` - PQE queries may need optimization 4. For `start_query_cost` high: check locks or resource contention 5. For `get_next_cost` high: optimize SQL or add indexes 6. Regular cleanup: set appropriate retention period

path: VERSION
content: 0.2.0


path: package.json
content: {
  "name": "hologres-slow-query-analysis",
  "version": "1.0.1",
  "description": "Hologres slow query log analysis and diagnosis skill. Use for analyzing slow queries, failed queries, query performance diagnosis, and log management in Alibaba Cloud Hologres. Triggers: \"hologres slow query\", \"hg_query_log\", \"query diagnosis\", \"慢Query分析\", \"Hologres性能诊断\"",
  "publishConfig": {
    "registry": "https://contextlab.alibaba-inc.com/skill"
  },
  "aoneKit": {
    "generated": true
  }
}


path: references/configuration.md
content: # Hologres Slow Query Log Configuration

Configuration parameters for controlling slow query log behavior.

## log_min_duration_statement

Controls the minimum query duration to be logged.

| Property | Value |
|----------|-------|
| Default | 100ms (V2.2.7+), 1s (earlier versions) |
| Minimum | 100ms |
| Scope | DB level (superuser), Session level (all users) |

### Usage

```sql
-- DB level (requires superuser)
ALTER DATABASE dbname SET log_min_duration_statement = '250ms';

-- Session level (all users)
SET log_min_duration_statement = '250ms';

-- Disable logging
SET log_min_duration_statement = '-1';
```

### Notes
- Only affects new queries, existing logs remain unchanged
- Logs all DDL statements regardless of this setting
- Logs all failed queries regardless of this setting

## log_min_duration_query_stats

Controls recording of query execution statistics.

| Property | Value |
|----------|-------|
| Default | 10s |
| Scope | DB level, Session level |

### Usage

```sql
-- DB level
ALTER DATABASE dbname SET log_min_duration_query_stats = '20s';

-- Session level
SET log_min_duration_query_stats = '20s';

-- Disable
SET log_min_duration_query_stats = '-1';
```

### Notes
- High storage consumption
- Lower values may slow down log queries
- Adjust to lower values only when troubleshooting

## log_min_duration_query_plan

Controls recording of query execution plans.

| Property | Value |
|----------|-------|
| Default | 10s |
| Scope | DB level, Session level |

### Usage

```sql
-- DB level
ALTER DATABASE dbname SET log_min_duration_query_plan = '10s';

-- Session level
SET log_min_duration_query_plan = '10s';

-- Disable
SET log_min_duration_query_plan = '-1';
```

### Notes
- Execution plans can be obtained via EXPLAIN anytime
- Usually not necessary to record

## hg_query_log_retention_time_sec (V3.0.27+)

Controls log retention period.

| Property | Value |
|----------|-------|
| Default | 30 days (2592000 seconds) |
| Range | 3-30 days (259200-2592000 seconds) |
| Scope | DB level |

### Usage

```sql
-- Set to 7 days
ALTER DATABASE dbname SET hg_query_log_retention_time_sec = 604800;

-- Set to 30 days (default)
ALTER DATABASE dbname SET hg_query_log_retention_time_sec = 2592000;
```

### Notes
- Only affects new logs
- Expired logs are cleaned immediately (not async)
- Only applies to new connections

## V1.1 Compatibility Fix

For V1.1.36-V1.1.49, enable complete statistics display:

```sql
-- DB level (recommended, one-time setting)
ALTER DATABASE db_name SET hg_experimental_force_sync_collect_execution_statistics = ON;

-- Session level
SET hg_experimental_force_sync_collect_execution_statistics = ON;
```


path: references/diagnostic-queries.md
content: # Hologres Diagnostic SQL Queries

Complete collection of diagnostic SQL queries for `hologres.hg_query_log`.

## User Statistics

### Query count per user
```sql
SELECT usename AS "User",
       count(1) as "Query Count"
FROM hologres.hg_query_log
GROUP BY usename
ORDER BY count(1) DESC;
```

## Query Details

### Get specific query by ID
```sql
SELECT * FROM hologres.hg_query_log WHERE query_id = '13001450118416xxxx';
```

## New Query Analysis

### Count new queries yesterday
```sql
SELECT COUNT(1)
FROM (
    SELECT DISTINCT t1.digest
    FROM hologres.hg_query_log t1
    WHERE t1.query_start >= CURRENT_DATE - INTERVAL '1 day'
      AND t1.query_start < CURRENT_DATE
      AND NOT EXISTS (
          SELECT 1 FROM hologres.hg_query_log t2
          WHERE t2.digest = t1.digest
            AND t2.query_start < CURRENT_DATE - INTERVAL '1 day'
      )
      AND digest IS NOT NULL
) AS a;
```

### New queries by type
```sql
SELECT a.command_tag, COUNT(1)
FROM (
    SELECT DISTINCT t1.digest, t1.command_tag
    FROM hologres.hg_query_log t1
    WHERE t1.query_start >= CURRENT_DATE - INTERVAL '1 day'
      AND t1.query_start < CURRENT_DATE
      AND NOT EXISTS (
          SELECT 1 FROM hologres.hg_query_log t2
          WHERE t2.digest = t1.digest
            AND t2.query_start < CURRENT_DATE - INTERVAL '1 day'
      )
      AND t1.digest IS NOT NULL
) AS a
GROUP BY 1 ORDER BY 2 DESC;
```

### New query details
```sql
SELECT a.usename, a.status, a.query_id, a.digest, a.datname,
       a.command_tag, a.query, a.cpu_time_ms, a.memory_bytes
FROM (
    SELECT DISTINCT t1.usename, t1.status, t1.query_id, t1.digest,
           t1.datname, t1.command_tag, t1.query, t1.cpu_time_ms, t1.memory_bytes
    FROM hologres.hg_query_log t1
    WHERE t1.query_start >= CURRENT_DATE - INTERVAL '1 day'
      AND t1.query_start < CURRENT_DATE
      AND NOT EXISTS (
          SELECT 1 FROM hologres.hg_query_log t2
          WHERE t2.digest = t1.digest
            AND t2.query_start < CURRENT_DATE - INTERVAL '1 day'
      )
      AND t1.digest IS NOT NULL
) AS a;
```

### New query trend (hourly)
```sql
SELECT to_char(a.query_start, 'HH24') AS query_start_hour,
       a.command_tag, COUNT(1)
FROM (
    SELECT DISTINCT t1.query_start, t1.digest, t1.command_tag
    FROM hologres.hg_query_log t1
    WHERE t1.query_start >= CURRENT_DATE - INTERVAL '1 day'
      AND t1.query_start < CURRENT_DATE
      AND NOT EXISTS (
          SELECT 1 FROM hologres.hg_query_log t2
          WHERE t2.digest = t1.digest
            AND t2.query_start < CURRENT_DATE - INTERVAL '1 day'
      )
      AND t1.digest IS NOT NULL
) AS a
GROUP BY 1, 2 ORDER BY 3 DESC;
```

## Query Aggregation

### Slow query frequency by digest
```sql
SELECT digest, command_tag, count(1)
FROM hologres.hg_query_log
WHERE query_start >= CURRENT_DATE - INTERVAL '1 day'
  AND query_start < CURRENT_DATE
GROUP BY 1, 2
ORDER BY 3 DESC;
```

### Top 10 memory consumers (past week)
```sql
SELECT digest, avg(memory_bytes)
FROM hologres.hg_query_log
WHERE query_start >= CURRENT_DATE - INTERVAL '7 day'
  AND query_start < CURRENT_DATE
  AND digest IS NOT NULL
  AND memory_bytes IS NOT NULL
GROUP BY 1
ORDER BY 2 DESC
LIMIT 10;
```

## Traffic Analysis

### Hourly access volume (last 3 hours)
```sql
SELECT date_trunc('hour', query_start) AS query_start,
       count(1) AS query_count,
       sum(read_bytes) AS read_bytes,
       sum(cpu_time_ms) AS cpu_time_ms
FROM hologres.hg_query_log
WHERE query_start >= now() - interval '3 h'
GROUP BY 1;
```

## High Resource Period Analysis

### SQL template CPU consumption in specific time range
```sql
SELECT digest, count(1), avg(cpu_time_ms), sum(cpu_time_ms)
FROM hologres.hg_query_log
WHERE query_start > '2025-02-26 10:00:00+08'
  AND query_start < '2025-02-26 10:05:00+08'
GROUP BY digest
ORDER BY sum(cpu_time_ms) DESC
LIMIT 100;
```

### Comprehensive slow SQL analysis by time range
```sql
SELECT command_tag, digest, max(engine_type::text), count(1) total_count,
       sum(case when status != 'SUCCESS' then 1 else 0 end) fail_count,
       max(cpu_time_ms) max_cpu, avg(cpu_time_ms)::int avg_cpu,
       sum(cpu_time_ms) total_cpu, avg(duration)::int avg_duration,
       sum(duration) total_duration,
       max(case when status = 'SUCCESS' then query_id else null end) query_id,
       min(query_start)
FROM hologres.hg_query_log
WHERE query_start > '2025-12-09 14:10:00+08'
  AND query_start <= '2025-12-09 14:20:00+08'
  AND engine_type::text != '{FixedQE}'
GROUP BY 1, 2
ORDER BY sum(cpu_time_ms) DESC NULLS LAST
LIMIT 10;
```

## SQL Fingerprint (digest)

SQL fingerprint calculation rules:
- Only for SELECT, INSERT, DELETE, UPDATE queries
- Case handling follows Hologres SQL rules
- Ignores whitespace (spaces, newlines, tabs)
- Ignores specific constant values
- Array element count doesn't affect fingerprint
- Schema is auto-completed for table names


path: references/log-export.md
content: # Hologres Slow Query Log Export

Export slow query logs to Hologres internal tables or external tables (MaxCompute, OSS).

## Prerequisites

- Account must have access to `hologres.hg_query_log` table
- Superuser or pg_read_all_stats permission for full instance logs
- Use `query_start` as filter for better performance

## Important Notes

1. Use `query_start` column for time range filtering (indexed)
2. Don't wrap `query_start` in expressions (breaks index usage)
   - Bad: `WHERE to_char(query_start, 'yyyymmdd') = '20220101'`
   - Good: `WHERE query_start >= '2022-01-01' AND query_start < '2022-01-02'`

## Export to Hologres Internal Table

### 1. Create target table

```sql
CREATE TABLE query_log_download (
    usename text,
    status text,
    query_id text,
    datname text,
    command_tag text,
    duration integer,
    message text,
    query_start timestamp with time zone,
    query_date text,
    query text,
    result_rows bigint,
    result_bytes bigint,
    read_rows bigint,
    read_bytes bigint,
    affected_rows bigint,
    affected_bytes bigint,
    memory_bytes bigint,
    shuffle_bytes bigint,
    cpu_time_ms bigint,
    physical_reads bigint,
    pid integer,
    application_name text,
    engine_type text[],
    client_addr text,
    table_write text,
    table_read text[],
    session_id text,
    session_start timestamp with time zone,
    trans_id text,
    command_id text,
    optimization_cost integer,
    start_query_cost integer,
    get_next_cost integer,
    extended_cost text,
    plan text,
    statistics text,
    visualization_info text,
    query_detail text,
    query_extinfo text[]
);
```

### 2. Export data

```sql
INSERT INTO query_log_download
SELECT usename, status, query_id, datname, command_tag, duration,
       message, query_start, query_date, query, result_rows, result_bytes,
       read_rows, read_bytes, affected_rows, affected_bytes, memory_bytes,
       shuffle_bytes, cpu_time_ms, physical_reads, pid, application_name,
       engine_type, client_addr, table_write, table_read, session_id,
       session_start, trans_id, command_id, optimization_cost,
       start_query_cost, get_next_cost, extended_cost, plan, statistics,
       visualization_info, query_detail, query_extinfo
FROM hologres.hg_query_log
WHERE query_start >= '2022-08-03' AND query_start < '2022-08-04';
```

## Export to MaxCompute

### 1. Create MaxCompute table

```sql
-- Run in MaxCompute
CREATE TABLE IF NOT EXISTS mc_holo_query_log (
    username STRING COMMENT 'Query user',
    status STRING COMMENT 'Query status',
    query_id STRING COMMENT 'Query ID',
    datname STRING COMMENT 'Database name',
    command_tag STRING COMMENT 'Query type',
    duration BIGINT COMMENT 'Duration (ms)',
    message STRING COMMENT 'Error message',
    query STRING COMMENT 'Query text',
    read_rows BIGINT COMMENT 'Rows read',
    read_bytes BIGINT COMMENT 'Bytes read',
    memory_bytes BIGINT COMMENT 'Memory used',
    shuffle_bytes BIGINT COMMENT 'Shuffle bytes',
    cpu_time_ms BIGINT COMMENT 'CPU time (ms)',
    physical_reads BIGINT COMMENT 'Physical reads',
    application_name STRING COMMENT 'Application',
    engine_type ARRAY<STRING> COMMENT 'Query engine',
    table_write STRING COMMENT 'Write table',
    table_read ARRAY<STRING> COMMENT 'Read tables',
    plan STRING COMMENT 'Query plan',
    optimization_cost BIGINT COMMENT 'Plan time',
    start_query_cost BIGINT COMMENT 'Startup time',
    get_next_cost BIGINT COMMENT 'Execution time',
    extended_cost STRING COMMENT 'Other costs',
    query_detail STRING COMMENT 'Extra info (JSON)',
    query_extinfo ARRAY<STRING> COMMENT 'Extra info (Array)',
    query_start STRING COMMENT 'Start time',
    query_date STRING COMMENT 'Start date'
) COMMENT 'hologres instance query log daily'
PARTITIONED BY (ds STRING COMMENT 'stat date')
LIFECYCLE 365;

ALTER TABLE mc_holo_query_log ADD PARTITION (ds=20220803);
```

### 2. Import foreign schema and export

```sql
-- Run in Hologres
IMPORT FOREIGN SCHEMA project_name LIMIT TO (mc_holo_query_log)
FROM SERVER odps_server INTO public;

INSERT INTO mc_holo_query_log
SELECT usename AS username, status, query_id, datname, command_tag,
       duration, message, query, read_rows, read_bytes, memory_bytes,
       shuffle_bytes, cpu_time_ms, physical_reads, application_name,
       engine_type, table_write, table_read, plan, optimization_cost,
       start_query_cost, get_next_cost, extended_cost, query_detail,
       query_extinfo, query_start, query_date, '20220803'
FROM hologres.hg_query_log
WHERE query_start >= '2022-08-03' AND query_start < '2022-08-04';
```


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)