Skill: wenbingyu/hologres-schema-generator (version 0.2.0)

--- name: hologres-schema-generator description: | Hologres DDL schema design and table creation expert. Use for generating CREATE TABLE statements, choosing storage formats (column/row/row-column), configuring table properties (distribution_key, clustering_key, bitmap_columns, event_time_column), designing partition tables, selecting data types, and optimizing table schemas for different query patterns. Triggers: "hologres建表", "hologres create table", "hologres schema", "hologres DDL", "distribution_key", "clustering_key", "bitmap_columns", "event_time_column", "列存", "行存", "行列共存", "分区表", "hologres table design", "表设计", "hologres数据类型" --- ## Prerequisites This skill requires **hologres-cli** to be installed first: ```bash pip install hologres-cli export HOLOGRES_SKILL=hologres-schema-generator ``` All SQL execution depends on `hologres-cli` commands (`hologres sql run --write`, `hologres table create`). # Hologres Schema Design & Table Creation Generate optimized Hologres DDL statements with proper storage format, indexing, and distribution strategies. ## Information Gathering (IMPORTANT) Before generating DDL, check whether the user has provided enough context. Many Hologres table properties are **immutable after creation** (orientation, distribution_key, clustering_key, event_time_column, primary key, partition column) — getting them wrong means recreating the table. ### Required Information If ANY of the following are unclear, **ask the user before generating DDL**: | Dimension | Why Critical | Example Question | |-----------|-------------|-----------------| | **Query pattern** | Determines `orientation` (column/row/row-column) | "This table is mainly used for OLAP analytics, KV point lookups, or both?" | | **Column definitions** | Core schema structure | "Can you list the main columns and their rough types (text, number, timestamp, etc.)?" | | **Primary key / unique key** | Determines UPSERT capability; immutable | "Does this table need deduplication or updates by a unique key? If so, which column(s)?" | | **Main query conditions** | Determines distribution_key, clustering_key, bitmap | "What columns are most often used in WHERE, JOIN, or GROUP BY?" | ### Conditionally Required Ask these only when relevant signals appear: | Signal | Follow-up Question | |--------|-------------------| | Table looks like time-series or log data | "What is the estimated daily data volume? (determines whether to use partitioning)" | | User mentions JOIN with other tables | "Which column will this table JOIN on, and what is the distribution_key of the other table?" | | Large data volume or mentions retention | "How long should data be retained? (recommend dynamic partition management for lifecycle)" | | User mentions both analytics and serving | "What is the read pattern — batch scans, point lookups by ID, or both?" | ### Can Use Defaults (No Need to Ask) | Property | Default Behavior | |----------|-----------------| | `bitmap_columns` | Auto-enabled for TEXT columns; can ALTER later | | `dictionary_encoding_columns` | Use `:auto` mode; can ALTER later | | `storage_mode` | Default `hot`; can ALTER later | | `time_to_live_in_seconds` | **Not recommended.** Deletion time is non-deterministic. Use dynamic partition management instead | ### Gathering Flow ``` User request → Check available info | +-- Columns + query pattern + key info all clear? | → Generate DDL directly | +-- Missing critical info? | → Ask 1-3 focused questions (batch them, don't ask one at a time) | +-- Very vague request (e.g., "help me create a user table")? → Ask: 1) main columns 2) query pattern 3) key/dedup needs ``` > **Principle:** Batch questions into a single round. Never ask more than 3-4 questions at once. If the user provides partial info, fill in reasonable defaults for the rest and explain your assumptions in comments. ## Storage Formats Choose the storage format based on your primary query pattern. | Format | Syntax | Best For | Primary Key | |--------|--------|----------|-------------| | **Column store** | `orientation = 'column'` | OLAP, aggregation, scan-heavy queries | Optional | | **Row store** | `orientation = 'row'` | Point lookups, high-QPS KV queries | Required | | **Row-column store** | `orientation = 'row,column'` | Mixed workloads (OLAP + point lookup) | Required | > **Default recommendation:** Use `row,column` (row-column store) when the workload is unclear. It handles both OLAP and point queries well. ## Quick Start ```sql -- Column store table (OLAP / analytics) CREATE TABLE orders ( order_id BIGINT NOT NULL, user_id BIGINT, amount DECIMAL(18,2), status TEXT, order_time TIMESTAMPTZ NOT NULL, ds TEXT NOT NULL, PRIMARY KEY (order_id) ) PARTITION BY LIST (ds) WITH ( orientation = 'column', distribution_key = 'order_id', clustering_key = 'order_time:asc', event_time_column = 'order_time', bitmap_columns = 'status,user_id' ); -- Row store table (high-QPS point lookup) CREATE TABLE user_profile ( user_id BIGINT NOT NULL, name TEXT, email TEXT, tags JSONB, updated_at TIMESTAMPTZ, PRIMARY KEY (user_id) ) WITH ( orientation = 'row', distribution_key = 'user_id', clustering_key = 'user_id' ); ``` ## CREATE TABLE Syntax (V2.1+) ```sql CREATE TABLE [IF NOT EXISTS] [schema.]table_name ( column_name data_type [NOT NULL] [DEFAULT expr], ... [PRIMARY KEY (col1 [, col2, ...])] ) [PARTITION BY LIST (partition_column)] WITH ( orientation = '{column | row | row,column}', distribution_key = 'col1[,col2]', clustering_key = 'col1[:asc|:desc][,col2[:asc|:desc]]', event_time_column = 'col', bitmap_columns = 'col1[,col2,...]', dictionary_encoding_columns = 'col1[:auto|:on|:off][,...]', time_to_live_in_seconds = 'N', -- NOT recommended, see below storage_mode = '{hot | cold}', table_group = 'group_name' ); ``` ### Legacy Syntax (All Versions) ```sql BEGIN; CREATE TABLE table_name (...); CALL set_table_property('table_name', 'orientation', 'column'); CALL set_table_property('table_name', 'distribution_key', 'col1'); CALL set_table_property('table_name', 'clustering_key', 'col1:asc'); CALL set_table_property('table_name', 'event_time_column', 'col1'); CALL set_table_property('table_name', 'bitmap_columns', 'col1,col2'); COMMIT; ``` ## Table Properties Reference ### distribution_key (Distribution Key) Controls how data is hash-distributed across shards. Critical for JOIN and GROUP BY performance. | Rule | Description | |------|-------------| | Choose high-cardinality columns | Avoid data skew | | Use JOIN/GROUP BY columns | Enable local computation, avoid shuffle | | Max 2 columns | More columns reduce distribution effectiveness | | Must be subset of PK | If table has a primary key | | Immutable after creation | Cannot ALTER, must recreate table | ```sql -- Single column distribution_key = 'user_id' -- Two columns (join on both) distribution_key = 'order_id,user_id' ``` ### clustering_key (Clustering Key / Sorted Index) Physically sorts data within files. Accelerates range queries and filters. | Rule | Description | |------|-------------| | Use range-query columns | e.g., timestamp, date columns | | Max 2-3 columns | More columns dilute sort benefit | | Put high-selectivity column first | Most-filtered column goes first | | Specify sort order | `:asc` (default) or `:desc` | | Column store only | Row store uses PK as sort key | ```sql clustering_key = 'order_time:asc' clustering_key = 'ds:asc,order_time:asc' ``` ### event_time_column (Segment Key) Organizes data files by time ranges. Enables file-level pruning for time-range queries. | Rule | Description | |------|-------------| | Use time/date columns | Timestamp of data ingestion or event time | | At most 1 column | Only one segment key per table | | Column store only | Not applicable to row store | | Combine with partition | Partition for coarse pruning, segment key for fine pruning | ```sql event_time_column = 'event_time' ``` ### bitmap_columns (Bitmap Index) Builds bitmap indexes for fast equality filtering on low-to-medium cardinality columns. | Rule | Description | |------|-------------| | Use filter columns | Columns frequently in WHERE clause | | Low-medium cardinality | status, type, region — NOT user_id | | TEXT columns auto-enabled | Default bitmap for text columns | | Can be added after creation | `ALTER TABLE ... SET (bitmap_columns = ...)` | ```sql bitmap_columns = 'status,payment_type,region' ``` ### dictionary_encoding_columns (Dictionary Encoding) Compresses text columns by mapping values to integers. Speeds up GROUP BY and aggregations. | Rule | Description | |------|-------------| | Use `:auto` mode | Let Hologres decide based on cardinality | | Good for low-cardinality text | country, status, category | | Avoid high-cardinality columns | user_id, order_id — no compression benefit | | Don't set if unsure | Incorrect setting may hurt performance | ```sql dictionary_encoding_columns = 'country:auto,status:auto' ``` ### time_to_live_in_seconds (TTL) — NOT RECOMMENDED > **Do NOT use `time_to_live_in_seconds`.** The actual deletion time is non-deterministic — data will be deleted at an arbitrary time after the specified TTL, not at a precise point. This makes it unreliable for data lifecycle management. > > **Recommended alternative:** Use **dynamic partition management** — create daily/hourly partitions and drop old partitions on a schedule (via cron or scheduling system). This gives you precise, predictable data lifecycle control. ```sql -- BAD: TTL-based lifecycle (deletion time unpredictable) -- time_to_live_in_seconds = '2592000' -- GOOD: Partition-based lifecycle (precise control) -- 1. Create table with daily partitions CREATE TABLE events ( event_id BIGINT NOT NULL, ds TEXT NOT NULL, PRIMARY KEY (ds, event_id) ) PARTITION BY LIST (ds) WITH (orientation = 'column', distribution_key = 'event_id'); -- 2. Drop old partitions on schedule (e.g., retain 30 days) DROP TABLE IF EXISTS events_20251101; -- drop partition older than 30 days ``` ## Primary Key Design | Storage Format | PK Requirement | Notes | |---------------|---------------|-------| | Column store | Optional | Add PK only if needed for UPSERT | | Row store | **Required** | PK drives the row-store index | | Row-column store | **Required** | PK serves both point lookup and analytics | **Rules:** - Max 32 columns in composite PK - PK columns must be NOT NULL and UNIQUE - Prohibited types: `FLOAT`, `DOUBLE`, `NUMERIC`, `ARRAY`, `JSON`, `JSONB` - PK cannot be altered after creation — must recreate table - Avoid `SERIAL` as PK — causes table-level locks on write ## Partition Table Design Use partitions for large tables with time-based or categorical data. ```sql -- Parent table CREATE TABLE events ( event_id BIGINT NOT NULL, user_id BIGINT, event_type TEXT, ds TEXT NOT NULL, PRIMARY KEY (ds, event_id) ) PARTITION BY LIST (ds) WITH ( orientation = 'column', distribution_key = 'event_id', clustering_key = 'ds:asc', event_time_column = 'ds' ); -- Child partitions CREATE TABLE events_20251201 PARTITION OF events FOR VALUES IN ('20251201'); CREATE TABLE events_20251202 PARTITION OF events FOR VALUES IN ('20251202'); ``` **Rules:** - Only `LIST` partitioning is supported - Partition column must be part of PK (if PK exists) - Supported partition column types: `TEXT`, `VARCHAR`, `INT`, `DATE` (V1.3.22+) - Skip daily partitions if daily data < 100M rows — use `event_time_column` instead - Always filter on partition column in queries for pruning ## Scenario-Based Templates ### 1. High-QPS Point Lookup (KV) ```sql CREATE TABLE user_kv ( user_id BIGINT NOT NULL PRIMARY KEY, profile JSONB, updated_at TIMESTAMPTZ ) WITH ( orientation = 'row', distribution_key = 'user_id' ); ``` ### 2. Prefix Range Scan ```sql CREATE TABLE order_lines ( order_id BIGINT NOT NULL, line_no INT NOT NULL, product_id BIGINT, quantity INT, PRIMARY KEY (order_id, line_no) ) WITH ( orientation = 'row', distribution_key = 'order_id', clustering_key = 'order_id' ); ``` ### 3. Time-Range Analytics (Partitioned) ```sql CREATE TABLE page_views ( view_id BIGINT NOT NULL, user_id BIGINT, page_url TEXT, view_time TIMESTAMPTZ, ds TEXT NOT NULL, PRIMARY KEY (ds, view_id) ) PARTITION BY LIST (ds) WITH ( orientation = 'column', distribution_key = 'view_id', clustering_key = 'view_time:asc', event_time_column = 'view_time', bitmap_columns = 'user_id,page_url' ); ``` ### 4. Dimension Filtering (Non-Time) ```sql CREATE TABLE product_stats ( product_id BIGINT NOT NULL PRIMARY KEY, category TEXT, brand TEXT, sales_count BIGINT, revenue DECIMAL(18,2) ) WITH ( orientation = 'column', distribution_key = 'product_id', clustering_key = 'category:asc', bitmap_columns = 'category,brand' ); ``` ### 5. JOIN-Optimized Fact + Dimension ```sql -- Fact table: distribute by join key CREATE TABLE fact_orders ( order_id BIGINT NOT NULL PRIMARY KEY, customer_id BIGINT, amount DECIMAL(18,2), order_date TEXT ) WITH ( orientation = 'column', distribution_key = 'customer_id' ); -- Dimension table: same distribution key CREATE TABLE dim_customers ( customer_id BIGINT NOT NULL PRIMARY KEY, name TEXT, region TEXT ) WITH ( orientation = 'row,column', distribution_key = 'customer_id' ); -- JOIN on customer_id → local join, no shuffle ``` ### 6. Mixed Workload (OLAP + Serving) ```sql CREATE TABLE realtime_metrics ( metric_id BIGINT NOT NULL, device_id BIGINT NOT NULL, value DOUBLE PRECISION, ts TIMESTAMPTZ NOT NULL, ds TEXT NOT NULL, PRIMARY KEY (ds, device_id, metric_id) ) PARTITION BY LIST (ds) WITH ( orientation = 'row,column', distribution_key = 'device_id', clustering_key = 'ts:asc', event_time_column = 'ts', bitmap_columns = 'device_id' ); ``` ## References | Document | Content | |----------|---------| | [data-types.md](references/data-types.md) | Complete data type reference | | [table-properties.md](references/table-properties.md) | Detailed table property guide with selection flowcharts | | [partition-guide.md](references/partition-guide.md) | Partition table design and management | ## Best Practices 1. **Choose storage format first** — column for OLAP, row for KV, row-column when unsure 2. **Set `distribution_key` to JOIN/GROUP BY columns** — avoids cross-shard shuffle 3. **Set `clustering_key` for range-query columns** — improves time-range and filter queries 4. **Set `event_time_column` for time-series data** — enables file-level pruning 5. **Use `bitmap_columns` for low-cardinality filter columns** — status, type, region 6. **Partition by date only if daily data > 100M rows** — otherwise use segment key alone 7. **PK must include partition column** — required by Hologres for partition tables 8. **Align `distribution_key` across JOINed tables** — same column enables local join 9. **Use `WITH` syntax (V2.1+)** — cleaner than `CALL set_table_property` in transactions 10. **Avoid `SERIAL` as primary key** — causes table-level write locks

path: VERSION
content: 0.2.0


path: package.json
content: {
  "name": "hologres-schema-generator",
  "version": "1.0.1",
  "description": "Hologres DDL schema design and table creation expert. Use for generating CREATE TABLE statements, choosing storage formats (column/row/row-column), configuring table properties (distribution_key, clustering_key, bitmap_columns, event_time_column), designing partition tables, selecting data types, and optimizing table schemas for different query patterns. Triggers: \"hologres建表\", \"hologres create table\", \"hologres schema\", \"hologres DDL\", \"distribution_key\", \"clustering_key\", \"bitmap_columns\", \"event_time_column\", \"列存\", \"行存\", \"行列共存\", \"分区表\", \"hologres table design\", \"表设计\", \"hologres数据类型\"",
  "publishConfig": {
    "registry": "https://contextlab.alibaba-inc.com/skill"
  },
  "aoneKit": {
    "generated": true
  }
}


path: references/data-types.md
content: # Hologres Data Types Reference

## Numeric Types

| Type | Alias | Size | Range | Notes |
|------|-------|------|-------|-------|
| `SMALLINT` | `INT2` | 2 bytes | -32768 ~ 32767 | |
| `INTEGER` | `INT`, `INT4` | 4 bytes | -2^31 ~ 2^31-1 | Most common integer |
| `BIGINT` | `INT8` | 8 bytes | -2^63 ~ 2^63-1 | Recommended for IDs |
| `SERIAL` | - | 4 bytes | Auto-increment | Avoid as PK (causes table lock) |
| `BIGSERIAL` | - | 8 bytes | Auto-increment | Avoid as PK (causes table lock) |
| `REAL` | `FLOAT4` | 4 bytes | 6 decimal digits precision | Cannot be PK |
| `DOUBLE PRECISION` | `FLOAT8` | 8 bytes | 15 decimal digits precision | Cannot be PK |
| `NUMERIC(p,s)` | `DECIMAL(p,s)` | Variable | p: 1~38, s: 0~p | Cannot be PK; best for monetary values |
| `MONEY` | - | 8 bytes | -922337203685477.5808 ~ 922337203685477.5807 | Two decimal places |
| `BOOLEAN` | `BOOL` | 1 byte | TRUE / FALSE / NULL | |

## String / Character Types

| Type | Description | Max Length | Notes |
|------|-------------|-----------|-------|
| `TEXT` | Variable-length string | Unlimited | Recommended for most text columns |
| `VARCHAR(n)` | Variable-length with limit | n characters | Functionally same as TEXT with check |
| `CHAR(n)` | Fixed-length, blank-padded | n characters | Space padded; rarely needed |

> **Recommendation:** Always use `TEXT` unless you have a specific reason for `VARCHAR(n)`.

## Date / Time Types

| Type | Size | Resolution | Range | Notes |
|------|------|-----------|-------|-------|
| `DATE` | 4 bytes | 1 day | 4713 BC ~ 5874897 AD | Date only |
| `TIMESTAMP` | 8 bytes | 1 microsecond | 4713 BC ~ 294276 AD | Without timezone |
| `TIMESTAMPTZ` | 8 bytes | 1 microsecond | 4713 BC ~ 294276 AD | **Recommended** for event time |
| `TIME` | 8 bytes | 1 microsecond | 00:00:00 ~ 24:00:00 | Time only, no date |
| `TIMETZ` | 12 bytes | 1 microsecond | 00:00:00+1559 ~ 24:00:00-1559 | Time with timezone |
| `INTERVAL` | 16 bytes | 1 microsecond | -178000000 ~ 178000000 years | Time interval |

> **Recommendation:** Use `TIMESTAMPTZ` instead of `TIMESTAMP` for event timestamps to avoid timezone issues.

## Binary Types

| Type | Description | Notes |
|------|-------------|-------|
| `BYTEA` | Variable-length binary data | For binary blobs, images, serialized objects |

## Network Types

| Type | Description | Notes |
|------|-------------|-------|
| `INET` | IPv4 / IPv6 host address | Supports CIDR notation |

## Identifier Types

| Type | Description | Notes |
|------|-------------|-------|
| `UUID` | 128-bit universally unique identifier | `gen_random_uuid()` to generate |
| `OID` | Object identifier | Internal PostgreSQL type |

## JSON Types

| Type | Description | Notes |
|------|-------------|-------|
| `JSON` | Text JSON | Stores as-is, re-parses on access |
| `JSONB` | Binary JSON | **Recommended**; indexed, faster queries |

```sql
-- JSONB usage examples
CREATE TABLE events (
  id BIGINT PRIMARY KEY,
  payload JSONB
) WITH (orientation = 'column');

-- Query nested fields
SELECT payload->>'user_id' FROM events WHERE payload->>'type' = 'click';
SELECT payload->'address'->>'city' FROM events;
```

> **Note:** JSONB cannot be used as PK, distribution_key, clustering_key, or partition column.

## Bit String Types

| Type | Description | Notes |
|------|-------------|-------|
| `BIT(n)` | Fixed-length bit string | Exactly n bits |
| `VARBIT(n)` | Variable-length bit string | Up to n bits |

## Array Types

Hologres supports one-dimensional arrays of the following base types:

| Array Type | Base Type |
|-----------|-----------|
| `INT4[]` | INTEGER |
| `INT8[]` | BIGINT |
| `FLOAT4[]` | REAL |
| `FLOAT8[]` | DOUBLE PRECISION |
| `BOOLEAN[]` | BOOLEAN |
| `TEXT[]` | TEXT |
| `NUMERIC[]` | NUMERIC |

```sql
-- Array column example
CREATE TABLE user_tags (
  user_id BIGINT NOT NULL PRIMARY KEY,
  tags TEXT[],
  scores FLOAT8[]
) WITH (orientation = 'column');

-- Query
SELECT * FROM user_tags WHERE 'vip' = ANY(tags);
SELECT * FROM user_tags WHERE tags @> ARRAY['vip','active'];
```

> **Note:** Array columns cannot be PK, distribution_key, clustering_key, or partition column.

## Extension Types

| Type | Extension | Description | Notes |
|------|-----------|-------------|-------|
| `RoaringBitmap` | `roaringbitmap` | Compressed bitmap | UV/PV dedup; requires `CREATE EXTENSION roaringbitmap` |
| `PostGIS types` | `postgis` | Geometry/Geography | Spatial queries; requires `CREATE EXTENSION postgis` |

## Data Type Selection Guide

### For Primary Key

| Scenario | Recommended Type |
|----------|-----------------|
| Numeric ID | `BIGINT` |
| Composite key (date + id) | `TEXT` + `BIGINT` |
| Natural key | `TEXT` or `VARCHAR` |

**Prohibited as PK:** `FLOAT`, `DOUBLE PRECISION`, `NUMERIC`, `ARRAY`, `JSON`, `JSONB`

### For Distribution Key

| Scenario | Recommended Type |
|----------|-----------------|
| User/Order ID | `BIGINT` |
| String-based ID | `TEXT` |

**Prohibited as distribution_key:** `FLOAT`, `DOUBLE PRECISION`, `ARRAY`, `JSON`, `JSONB`

### For Partition Column

| Scenario | Recommended Type |
|----------|-----------------|
| Date string (yyyyMMdd) | `TEXT` |
| Date type | `DATE` (V1.3.22+) |
| Category ID | `INT` |

**Supported partition types:** `TEXT`, `VARCHAR`, `INT`, `BIGINT`, `DATE`

### For Clustering Key

| Scenario | Recommended Type |
|----------|-----------------|
| Time range query | `TIMESTAMPTZ`, `DATE`, `TEXT` (date string) |
| Category filtering | `TEXT`, `INT` |

### For Bitmap Columns

| Good candidates | Bad candidates |
|----------------|---------------|
| status, type, region (low cardinality) | user_id, order_id (high cardinality) |
| TEXT columns (auto-enabled by default) | Numeric IDs |

## Type Conversion Functions

```sql
-- Common casts
column::BIGINT       -- to BIGINT
column::TEXT         -- to TEXT
column::TIMESTAMPTZ  -- to TIMESTAMPTZ
column::DATE         -- to DATE
column::JSONB        -- to JSONB

-- Explicit cast syntax
CAST(column AS BIGINT)
CAST(column AS TIMESTAMPTZ)
```


path: references/partition-guide.md
content: # Hologres Partition Table Guide

## Overview

Hologres supports **LIST partitioning** only. Partition tables divide data into child tables by discrete values (typically dates), enabling partition pruning in queries.

## Syntax

### Create Parent Table

```sql
-- V2.1+ WITH syntax
CREATE TABLE parent_table (
  id BIGINT NOT NULL,
  ds TEXT NOT NULL,
  ...,
  PRIMARY KEY (ds, id)
)
PARTITION BY LIST (ds)
WITH (
  orientation = 'column',
  distribution_key = 'id',
  clustering_key = 'id:asc',
  event_time_column = 'ds'
);

-- Legacy syntax
BEGIN;
CREATE TABLE parent_table (
  id BIGINT NOT NULL,
  ds TEXT NOT NULL,
  PRIMARY KEY (ds, id)
) PARTITION BY LIST (ds);
CALL set_table_property('parent_table', 'orientation', 'column');
CALL set_table_property('parent_table', 'distribution_key', 'id');
COMMIT;
```

### Create Child Partitions

```sql
-- Single value per partition (most common: daily partitions)
CREATE TABLE parent_20251201 PARTITION OF parent_table FOR VALUES IN ('20251201');
CREATE TABLE parent_20251202 PARTITION OF parent_table FOR VALUES IN ('20251202');

-- Multiple values per partition (e.g., weekly)
CREATE TABLE parent_w01 PARTITION OF parent_table FOR VALUES IN ('20251201','20251202','20251203','20251204','20251205','20251206','20251207');
```

### Dynamic Partition Creation (Writing)

If writing to a partition value that has no child table, the write will fail. You must create the partition before writing.

```sql
-- Auto-create partition on write (V2.0+)
-- Set at session level
SET hg_experimental_enable_create_partition_on_insert = ON;

-- Or at table level (V2.2+)
ALTER TABLE parent_table SET (auto_partitioning = 'true');
```

## Partition Column Rules

| Rule | Description |
|------|-------------|
| Only **LIST** type | RANGE and HASH partitioning not supported |
| **Must be in PK** | If the table has a primary key, partition column must be part of it |
| Only **1 column** | Cannot partition by multiple columns |
| Supported types | `TEXT`, `VARCHAR(n)`, `INT`, `BIGINT`, `DATE` (V1.3.22+) |
| Cannot be altered | Partition column is set at creation, cannot change |

## When to Use Partitions

### Decision Flowchart

```
Daily data volume?
  |
  +-- < 100M rows/day --> No partition. Use event_time_column only
  |
  +-- >= 100M rows/day
       |
       +-- Queries always filter by date? --> Partition by date (TEXT ds)
       |
       +-- Queries filter by category? --> Partition by category (if few values)
       |
       +-- No clear filter pattern --> No partition. Use clustering_key
```

### Good Partition Candidates

| Scenario | Partition Column | Type | Partition Granularity |
|----------|-----------------|------|----------------------|
| Daily log analytics | `ds` | TEXT | Daily: '20251201' |
| Monthly billing | `month` | TEXT | Monthly: '202512' |
| Regional data | `region` | TEXT | Per region: 'cn','us','eu' |
| Date-typed events | `event_date` | DATE | Daily |

### Bad Partition Candidates

| Scenario | Why Not |
|----------|---------|
| Small daily data (< 100M rows) | Too many small files, hurts scan performance |
| High-cardinality column (user_id) | Creates millions of tiny partitions |
| Frequently changing filter patterns | Queries may miss partition pruning |
| No common filter column | No pruning benefit |

## Partition with Table Properties

All table properties (distribution_key, clustering_key, etc.) are set on the **parent table** and inherited by all child partitions.

```sql
CREATE TABLE events (
  event_id BIGINT NOT NULL,
  user_id BIGINT,
  event_type TEXT,
  event_time TIMESTAMPTZ,
  ds TEXT NOT NULL,
  PRIMARY KEY (ds, event_id)
)
PARTITION BY LIST (ds)
WITH (
  orientation = 'column',
  distribution_key = 'event_id',
  clustering_key = 'event_time:asc',
  event_time_column = 'event_time',
  bitmap_columns = 'event_type,user_id'
);
```

## Partition Management

### List Partitions

```sql
-- Show all child partitions
SELECT
  nmsp_parent.nspname AS parent_schema,
  parent.relname AS parent_table,
  nmsp_child.nspname AS child_schema,
  child.relname AS child_table
FROM pg_inherits
  JOIN pg_class parent ON pg_inherits.inhparent = parent.oid
  JOIN pg_class child ON pg_inherits.inhrelid = child.oid
  JOIN pg_namespace nmsp_parent ON parent.relnamespace = nmsp_parent.oid
  JOIN pg_namespace nmsp_child ON child.relnamespace = nmsp_child.oid
WHERE parent.relname = 'parent_table';
```

### Drop Partition

```sql
-- Drop a specific child partition (data is deleted)
DROP TABLE parent_20251201;

-- Detach partition (preserves data as standalone table)
ALTER TABLE parent_table DETACH PARTITION parent_20251201;
```

### Rename Partition

```sql
ALTER TABLE parent_20251201 RENAME TO parent_20251201_backup;
```

### Attach Existing Table as Partition

```sql
-- Existing table must have compatible schema
ALTER TABLE parent_table ATTACH PARTITION existing_table FOR VALUES IN ('20251210');
```

## Partition-Based Data Lifecycle (Recommended)

> **Do NOT use `time_to_live_in_seconds` for data lifecycle.** TTL deletion is non-deterministic — data is deleted at an arbitrary time after the specified duration, making it unreliable.
>
> **Use partition-based lifecycle instead** — drop old partitions on a schedule for precise, predictable data management.

### Lifecycle via Scheduled Partition Dropping

```sql
-- Create table with daily partitions (no TTL)
CREATE TABLE logs (
  log_id BIGINT NOT NULL,
  message TEXT,
  ds TEXT NOT NULL,
  log_time TIMESTAMPTZ,
  PRIMARY KEY (ds, log_id)
)
PARTITION BY LIST (ds)
WITH (
  orientation = 'column',
  distribution_key = 'log_id',
  event_time_column = 'log_time'
);

-- Drop partitions older than 30 days (run via cron / scheduling system)
-- Example: on 2025-12-31, drop partition for 2025-12-01
DROP TABLE IF EXISTS logs_20251201;
```

### Automation Script: Drop Old Partitions

```sql
-- Drop partitions older than N days
DO $$
DECLARE
  cutoff_date DATE := CURRENT_DATE - INTERVAL '30 days';
  partition_value TEXT;
  partition_name TEXT;
  rec RECORD;
BEGIN
  FOR rec IN
    SELECT child.relname AS child_table
    FROM pg_inherits
      JOIN pg_class parent ON pg_inherits.inhparent = parent.oid
      JOIN pg_class child ON pg_inherits.inhrelid = child.oid
    WHERE parent.relname = 'logs'
  LOOP
    -- Extract date from partition name (assumes suffix is YYYYMMDD)
    partition_value := RIGHT(rec.child_table, 8);
    BEGIN
      IF TO_DATE(partition_value, 'YYYYMMDD') < cutoff_date THEN
        EXECUTE format('DROP TABLE IF EXISTS %I', rec.child_table);
        RAISE NOTICE 'Dropped partition: %', rec.child_table;
      END IF;
    EXCEPTION WHEN OTHERS THEN
      -- Skip partitions with non-date suffixes
      NULL;
    END;
  END LOOP;
END $$;
```

## Querying Partition Tables

### Partition Pruning

Always include partition column in WHERE clause for efficient queries:

```sql
-- Good: partition pruning occurs
SELECT * FROM events WHERE ds = '20251201' AND event_type = 'click';

-- Good: range pruning (scans only matching partitions)
SELECT * FROM events WHERE ds >= '20251201' AND ds <= '20251207';

-- Bad: full scan across ALL partitions
SELECT * FROM events WHERE event_type = 'click';
-- (no ds filter -> scans every partition)
```

### Query Child Partition Directly

```sql
-- Query specific partition (avoids pruning overhead for known partition)
SELECT * FROM events_20251201 WHERE event_type = 'click';
```

### INSERT / UPSERT

```sql
-- Insert into parent table (routes to correct partition)
INSERT INTO events (event_id, user_id, event_type, event_time, ds)
VALUES (1, 100, 'click', NOW(), '20251201');

-- Insert into child partition directly
INSERT INTO events_20251201 (event_id, user_id, event_type, event_time, ds)
VALUES (1, 100, 'click', NOW(), '20251201');

-- UPSERT (INSERT ON CONFLICT) works on both parent and child
INSERT INTO events VALUES (1, 100, 'click', NOW(), '20251201')
ON CONFLICT (ds, event_id) DO UPDATE SET event_type = EXCLUDED.event_type;
```

## Best Practices

1. **Use TEXT for partition column** -- Simplest and most compatible (ds = '20251201')
2. **One day per partition for daily data** -- Standard pattern for large datasets
3. **Pre-create partitions** -- Create future partitions in advance to avoid write failures
4. **Always filter on partition column** -- Prevents expensive full scans
5. **Combine with event_time_column** -- Partition for coarse pruning, segment for fine pruning
6. **Monitor partition count** -- Too many partitions (>1000) can degrade metadata operations
7. **Use scheduled partition dropping for lifecycle** -- Drop old partitions via cron; do NOT use `time_to_live_in_seconds`
8. **Partition column in PK** -- Required rule; put partition column first in PK for clarity
9. **Avoid partitioning small tables** -- Overhead exceeds benefit when daily data is small
10. **Same distribution_key across related tables** -- Enables local joins across partitioned tables

## Partition Automation Script

```sql
-- Create daily partitions for next 7 days
DO $$
DECLARE
  d DATE;
  partition_name TEXT;
  partition_value TEXT;
BEGIN
  FOR i IN 0..6 LOOP
    d := CURRENT_DATE + i;
    partition_value := TO_CHAR(d, 'YYYYMMDD');
    partition_name := 'events_' || partition_value;
    EXECUTE format(
      'CREATE TABLE IF NOT EXISTS %I PARTITION OF events FOR VALUES IN (%L)',
      partition_name, partition_value
    );
  END LOOP;
END $$;
```


path: references/table-properties.md
content: # Hologres Table Properties Reference

## Properties Overview

| Property | Column Store | Row Store | Row-Column Store | Alterable |
|----------|:----------:|:---------:|:----------------:|:---------:|
| `orientation` | Y | Y | Y | No |
| `distribution_key` | Y | Y | Y | No |
| `clustering_key` | Y | Ignored (uses PK) | Column part only | No |
| `event_time_column` | Y | Ignored | Column part only | No |
| `bitmap_columns` | Y | Ignored | Column part only | Yes |
| `dictionary_encoding_columns` | Y | Ignored | Column part only | Yes |
| `time_to_live_in_seconds` | Y | Y | Y | Yes |
| `storage_mode` | Y | Y | Y | Yes |
| `table_group` | Y | Y | Y | No |

## orientation (Storage Format)

Controls the physical storage layout of the table.

```sql
-- V2.1+ WITH syntax
WITH (orientation = 'column')
WITH (orientation = 'row')
WITH (orientation = 'row,column')

-- Legacy syntax
CALL set_table_property('table_name', 'orientation', 'column');
```

### Selection Flowchart

```
Query pattern?
  |
  +-- Mainly point lookups (WHERE pk = ?) --> 'row'
  |
  +-- Mainly OLAP (GROUP BY, aggregation, scan) --> 'column'
  |
  +-- Both point lookups AND analytics --> 'row,column'
  |
  +-- Unsure --> 'row,column' (safe default)
```

### Comparison

| Feature | Column Store | Row Store | Row-Column Store |
|---------|:----------:|:---------:|:----------------:|
| Full scan / aggregation | Fast | Slow | Fast |
| Point lookup by PK | Slow | Fast | Fast |
| Write throughput | High | Medium | Medium |
| Storage cost | Low (compressed) | Medium | Higher (stores both) |
| Primary key | Optional | Required | Required |
| Supports clustering_key | Yes | No (uses PK) | Yes (column part) |
| Supports bitmap_columns | Yes | No | Yes (column part) |

## distribution_key (Distribution Key)

Controls how rows are hash-distributed across shards.

```sql
-- Single column
WITH (distribution_key = 'user_id')

-- Two columns
WITH (distribution_key = 'order_id,user_id')

-- Legacy
CALL set_table_property('table_name', 'distribution_key', 'user_id');
```

### Selection Rules

1. **Choose JOIN key** -- Ensures co-located joins (local join, no shuffle)
2. **Choose GROUP BY key** -- Enables local aggregation
3. **High cardinality** -- Avoids data skew across shards
4. **Max 1-2 columns** -- More columns weaken distribution
5. **Must be subset of PK** -- If table has a primary key
6. **Immutable** -- Cannot change after table creation

### Common Patterns

| Scenario | distribution_key | Reason |
|----------|-----------------|--------|
| User table | `user_id` | Unique per user |
| Order table, joins user | `user_id` | Co-locate with user table |
| Event log | `event_id` or `user_id` | Even distribution or join alignment |
| Two-table join on A.x = B.x | Both use `x` | Local join |

### Anti-Patterns

| Bad Choice | Problem |
|-----------|---------|
| Low-cardinality column (status, gender) | Severe data skew |
| Timestamp column | All concurrent writes hit same shard |
| Too many columns (3+) | Hash distribution becomes random |
| Different keys for joined tables | Forces shuffle join |

## clustering_key (Clustering Key / Sorted Index)

Physically sorts data files. Accelerates range scans and range filters.

```sql
-- Ascending (default)
WITH (clustering_key = 'order_time:asc')

-- Multi-column with explicit order
WITH (clustering_key = 'ds:asc,order_time:desc')

-- Legacy
CALL set_table_property('table_name', 'clustering_key', 'order_time:asc');
```

### Selection Rules

1. **Use range-query columns** -- Columns with `>`, `<`, `BETWEEN`, `ORDER BY`
2. **Put highest-selectivity column first** -- Most filtered column first
3. **Max 2-3 columns** -- Diminishing returns with more
4. **Column store only** -- Row store uses PK as sort key
5. **Specify sort order** -- `:asc` (default) or `:desc`
6. **Immutable** -- Cannot change after table creation

### Interaction with event_time_column

```
Query: WHERE event_time BETWEEN ... AND ... AND status = 'active'
        |
        +-- event_time_column = 'event_time'  -> file-level pruning (coarse)
        +-- clustering_key = 'event_time:asc'  -> within-file range scan (fine)
        +-- bitmap_columns = 'status'          -> bitmap filter within files
```

## event_time_column (Segment Key)

Organizes data files by time ranges. Enables file-level pruning before data is read.

```sql
WITH (event_time_column = 'event_time')

-- Legacy
CALL set_table_property('table_name', 'event_time_column', 'event_time');
```

### Selection Rules

1. **Must be time-correlated column** -- Increasing over time
2. **Only 1 column** -- Cannot set multiple
3. **Column store only** -- Ignored for row store
4. **Best with TIMESTAMPTZ or TEXT date** -- ds (yyyyMMdd) or timestamp
5. **Combine with partition** -- Partition for coarse, segment for fine pruning
6. **Immutable** -- Cannot change after table creation

### When to Use vs. Partition

| Condition | Recommendation |
|-----------|---------------|
| Daily data < 100M rows | `event_time_column` only (no partition) |
| Daily data >= 100M rows | Partition + `event_time_column` |
| Non-time-based data | Neither (use clustering_key for filtering) |

## bitmap_columns (Bitmap Index)

Builds bitmap indexes for fast equality filtering.

```sql
WITH (bitmap_columns = 'status,payment_type,region')

-- Alterable after creation
ALTER TABLE table_name SET (bitmap_columns = 'status,payment_type,region,channel');

-- Legacy
CALL set_table_property('table_name', 'bitmap_columns', 'status,payment_type,region');
```

### Selection Rules

1. **Equality filter columns** -- `WHERE status = 'active'`, `WHERE region IN (...)`
2. **Low-to-medium cardinality** -- status (5), region (50), category (200)
3. **Column store only** -- Ignored for row store
4. **TEXT columns auto-enabled** -- No need to explicitly set for TEXT
5. **Alterable** -- Can add/remove after creation
6. **Avoid high-cardinality** -- user_id, order_id -- no benefit, wastes storage

### Recommended Cardinality

| Cardinality Range | Suitability |
|------------------|-------------|
| < 100 | Excellent |
| 100 ~ 10,000 | Good |
| 10,000 ~ 100,000 | Marginal |
| > 100,000 | Not recommended |

## dictionary_encoding_columns (Dictionary Encoding)

Compresses text by mapping to integers. Speeds up GROUP BY and filter on encoded columns.

```sql
WITH (dictionary_encoding_columns = 'country:auto,status:auto')

-- Alterable after creation
ALTER TABLE table_name SET (dictionary_encoding_columns = 'country:auto,status:auto');

-- Legacy
CALL set_table_property('table_name', 'dictionary_encoding_columns', 'country:auto,status:auto');
```

### Modes

| Mode | Description |
|------|-------------|
| `:auto` | Hologres decides based on cardinality (recommended) |
| `:on` | Force enable |
| `:off` | Force disable |

### Selection Rules

1. **Use `:auto` mode** -- Safest choice, let engine decide
2. **Good for low-cardinality text** -- country, status, category
3. **Avoid for high-cardinality** -- No compression benefit
4. **Column store only** -- Ignored for row store
5. **Alterable** -- Can change after creation

## time_to_live_in_seconds (TTL) — NOT RECOMMENDED

> **Do NOT use `time_to_live_in_seconds`.** The actual deletion time is non-deterministic — data will be deleted at an arbitrary time after the specified TTL, not at a precise point. This makes it unreliable for any data lifecycle management scenario.
>
> **Recommended alternative:** Use **dynamic partition management** — create daily/hourly partitions and drop old partitions on a schedule (via cron or external scheduling system). This gives precise, predictable lifecycle control.

### Syntax (for reference only)

```sql
-- NOT recommended
WITH (time_to_live_in_seconds = '2592000')  -- 30 days

-- Alterable after creation
ALTER TABLE table_name SET (time_to_live_in_seconds = '604800');  -- 7 days
```

### Why NOT to Use

| Problem | Description |
|---------|-------------|
| Non-deterministic deletion | Data is deleted at an arbitrary time AFTER the TTL, not at the precise TTL point |
| No visibility | Cannot predict or observe when deletion will actually happen |
| No atomicity | Partial data may be deleted while other rows in the same time range remain |
| Poor for compliance | Cannot guarantee data retention or deletion deadlines |

### Recommended Alternative: Partition-Based Lifecycle

```sql
-- Create table with daily partitions
CREATE TABLE events (
  event_id BIGINT NOT NULL,
  ds TEXT NOT NULL,
  PRIMARY KEY (ds, event_id)
) PARTITION BY LIST (ds)
WITH (orientation = 'column', distribution_key = 'event_id');

-- Drop old partitions on schedule (precise, deterministic)
-- e.g., retain 30 days: drop partitions older than 30 days via cron
DROP TABLE IF EXISTS events_20251101;
```

## storage_mode (Storage Tiering)

Controls hot/cold data storage placement.

```sql
WITH (storage_mode = 'hot')

-- Alterable
ALTER TABLE table_name SET (storage_mode = 'cold');
```

| Mode | Description |
|------|-------------|
| `hot` | SSD storage (default, fast) |
| `cold` | Cheaper storage (slow, for archival) |

## table_group

Associates the table with a specific Table Group for resource isolation.

```sql
WITH (table_group = 'my_group')
```

> **Note:** Table Group must be created in advance by an admin. Immutable after table creation.

## Property Selection Cheat Sheet

```
Step 1: Choose orientation
  - OLAP → 'column'
  - KV lookups → 'row'
  - Both / unsure → 'row,column'

Step 2: Set distribution_key
  - JOIN key or GROUP BY key
  - High cardinality, 1-2 columns
  - Must be PK subset

Step 3: Set clustering_key (column store only)
  - Range query columns
  - Time columns with :asc
  - Max 2-3 columns

Step 4: Set event_time_column (column store only)
  - Time/date column that increases
  - Only if time-range queries are common

Step 5: Set bitmap_columns (column store only)
  - WHERE = filter columns
  - Low-medium cardinality

Step 6: Data lifecycle
  - Use partition-based lifecycle (drop old partitions on schedule)
  - Do NOT use time_to_live_in_seconds (non-deterministic deletion)

Step 7: Consider partition
  - Only if daily data > 100M rows
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)