Skill: wenbingyu/hologres-uv-compute (version 0.2.0)

--- name: hologres-uv-compute description: | Hologres UV/PV computation using Dynamic Tables and RoaringBitmap for real-time deduplication at scale. Use for building incremental UV/PV pipelines, RoaringBitmap-based user deduplication, flexible time-range UV aggregation, and text-to-int UID encoding for bitmap compatibility. Triggers: "hologres uv", "hologres pv", "roaringbitmap", "rb_build_agg", "rb_or_agg", "去重", "UV计算", "用户去重", "bitmap去重", "实时UV", "hg_id_encoding" --- ## Prerequisites This skill requires **hologres-cli** to be installed first: ```bash pip install hologres-cli export HOLOGRES_SKILL=hologres-uv-compute ``` All SQL execution and Dynamic Table operations depend on `hologres-cli` commands (`hologres sql run --write`, `hologres dt create`). # Hologres UV/PV Computation with Dynamic Table & RoaringBitmap Build real-time, incremental UV/PV computation pipelines using Dynamic Tables and RoaringBitmap in Hologres. This approach supports flexible time-range aggregation over billions of records with low latency. ## Why This Approach | Traditional COUNT DISTINCT | RoaringBitmap + Dynamic Table | |---------------------------|-------------------------------| | Full scan on every query | Pre-aggregated bitmaps, incremental refresh | | Slow with high-cardinality UIDs | Compressed bitmap, sub-second UV queries | | Cannot merge across time ranges | `RB_OR_AGG` merges bitmaps for any date range | | Heavy resource usage | Incremental computation, minimal resources | ## Quick Start ```sql -- 1. Enable RoaringBitmap extension CREATE EXTENSION IF NOT EXISTS roaringbitmap; -- 2. Create ODS detail table (source data) BEGIN; CREATE TABLE ods_app_detail ( uid int, country text, prov text, city text, ymd text NOT NULL ) LOGICAL PARTITION BY LIST (ymd); CALL set_table_property('ods_app_detail', 'orientation', 'column'); CALL set_table_property('ods_app_detail', 'distribution_key', 'uid'); CALL set_table_property('ods_app_detail', 'clustering_key', 'ymd'); CALL set_table_property('ods_app_detail', 'event_time_column', 'ymd'); CALL set_table_property('ods_app_detail', 'bitmap_columns', 'country,prov,city,ymd'); COMMIT; -- 3. Create DWS Dynamic Table (bitmap aggregation layer) CREATE DYNAMIC TABLE dt_dws_app_rb ( country, prov, city, rb_uid, pv, ymd ) LOGICAL PARTITION BY LIST (ymd) WITH ( freshness = '5 minutes', auto_refresh_mode = 'incremental', auto_refresh_partition_active_time = '2 days', partition_key_time_format = 'YYYYMMDD' ) AS SELECT country, prov, city, RB_BUILD_AGG(uid) AS rb_uid, COUNT(1) AS pv, ymd FROM ods_app_detail GROUP BY country, prov, city, ymd; -- 4. Query UV/PV for a single day SELECT country, prov, city, RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv, SUM(pv) AS pv FROM dt_dws_app_rb WHERE ymd = '20251223' GROUP BY country, prov, city; ``` ## Architecture Overview ``` ODS (Detail) DWS (Bitmap Aggregation) Query ┌─────────────┐ Dynamic ┌──────────────────────┐ ┌─────────────┐ │ods_app_detail│──Table────>│ dt_dws_app_rb │───>│ RB_OR_AGG │ │ uid, dims, │ incremental│ rb_uid (bitmap), │ │ + CARDINALITY│ │ ymd │ refresh │ pv, dims, ymd │ │ = UV for any │ └─────────────┘ └──────────────────────┘ │ time range │ └─────────────┘ ``` **Data flow:** 1. Raw events flow into `ods_app_detail` (partitioned by day) 2. Dynamic Table `dt_dws_app_rb` incrementally aggregates UIDs into bitmaps per dimension per day 3. Queries merge bitmaps across any date range using `RB_OR_AGG` for exact UV ## ODS Detail Table Design The source table stores raw event data, partitioned by date. ```sql BEGIN; CREATE TABLE ods_app_detail ( uid int, country text, prov text, city text, ymd text NOT NULL ) LOGICAL PARTITION BY LIST (ymd); CALL set_table_property('ods_app_detail', 'orientation', 'column'); CALL set_table_property('ods_app_detail', 'distribution_key', 'uid'); CALL set_table_property('ods_app_detail', 'clustering_key', 'ymd'); CALL set_table_property('ods_app_detail', 'event_time_column', 'ymd'); CALL set_table_property('ods_app_detail', 'bitmap_columns', 'country,prov,city,ymd'); COMMIT; ``` **Key design choices:** | Property | Value | Reason | |----------|-------|--------| | `orientation` | `column` | Columnar storage for analytical queries | | `distribution_key` | `uid` | Distribute by user for aggregation locality | | `clustering_key` | `ymd` | Optimize time-range scans | | `event_time_column` | `ymd` | Segment key for partition pruning | | `bitmap_columns` | dimension columns | Accelerate dimension filtering | ## DWS Dynamic Table (Bitmap Aggregation) The Dynamic Table pre-aggregates UIDs into RoaringBitmaps per dimension per day using incremental refresh. ```sql CREATE DYNAMIC TABLE dt_dws_app_rb ( country, prov, city, rb_uid, pv, ymd ) LOGICAL PARTITION BY LIST (ymd) WITH ( freshness = '5 minutes', auto_refresh_mode = 'incremental', auto_refresh_partition_active_time = '2 days', partition_key_time_format = 'YYYYMMDD' ) AS SELECT country, prov, city, RB_BUILD_AGG(uid) AS rb_uid, COUNT(1) AS pv, ymd FROM ods_app_detail GROUP BY country, prov, city, ymd; ``` **Key Dynamic Table parameters:** | Parameter | Value | Description | |-----------|-------|-------------| | `freshness` | `5 minutes` | Target data freshness | | `auto_refresh_mode` | `incremental` | Only compute new/changed data | | [REDACTED_SECRET] | `2 days` | Only auto-refresh recent 2 days | | `partition_key_time_format` | `YYYYMMDD` | Parse partition key as date | ### Refresh Historical Partitions Auto-refresh only covers active partitions. For historical data, trigger manually: ```sql -- Full refresh a specific partition REFRESH DYNAMIC TABLE dt_dws_app_rb PARTITION(20251201) WITH (refresh_mode = 'full'); -- Full refresh a date range (one by one) REFRESH DYNAMIC TABLE dt_dws_app_rb PARTITION(20251201) WITH (refresh_mode = 'full'); ``` ## UV/PV Queries ### Single Day ```sql SELECT country, prov, city, RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv, SUM(pv) AS pv FROM dt_dws_app_rb WHERE ymd = '20251223' GROUP BY country, prov, city; ``` ### Date Range (e.g., Monthly) ```sql SELECT country, prov, city, RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv, SUM(pv) AS pv FROM dt_dws_app_rb WHERE ymd >= '20251201' AND ymd <= '20251231' GROUP BY country, prov, city; ``` ### Global UV (All Dimensions) ```sql SELECT RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS total_uv, SUM(pv) AS total_pv FROM dt_dws_app_rb WHERE ymd >= '20251201' AND ymd <= '20251231'; ``` **How it works:** `RB_OR_AGG` merges daily bitmaps with bitwise OR — a user appearing on multiple days is counted only once. `RB_CARDINALITY` returns the count of distinct bits (= distinct users). ## Text UID Encoding When UIDs are `text` type (e.g., device IDs, UUIDs), they must be mapped to integers for RoaringBitmap. Two approaches: ### Approach 1: Mapping Table (Manual) ```sql -- Create mapping table BEGIN; CREATE TABLE uid_mapping ( uid text NOT NULL, uid_int32 serial, PRIMARY KEY (uid) ); CALL set_table_property('uid_mapping', 'orientation', 'row'); CALL set_table_property('uid_mapping', 'distribution_key', 'uid'); CALL set_table_property('uid_mapping', 'clustering_key', 'uid'); COMMIT; -- Use in Dynamic Table CREATE DYNAMIC TABLE dt_dws_app_rb (...) AS SELECT country, prov, city, RB_BUILD_AGG(m.uid_int32) AS uid_rb, COUNT(1) AS pv, ymd FROM ods_app_detail o JOIN uid_mapping m ON o.uid_text = m.uid GROUP BY country, prov, city, ymd; ``` ### Approach 2: hg_id_encoding (V4.1+, Recommended) Built-in function that auto-manages the mapping table. ```sql -- Create mapping table (single text PK + one serial column) BEGIN; CREATE TABLE uid_mapping ( uid text NOT NULL, uid_int32 serial, PRIMARY KEY (uid) ); CALL set_table_property('uid_mapping', 'orientation', 'row'); CALL set_table_property('uid_mapping', 'distribution_key', 'uid'); COMMIT; -- Use hg_id_encoding_int4 in Dynamic Table query CREATE DYNAMIC TABLE dt_dws_app_rb (...) AS SELECT country, prov, city, RB_BUILD_AGG(hg_id_encoding_int4(uid_text, 'uid_mapping')) AS uid_rb, COUNT(1) AS pv, ymd FROM ods_app_detail GROUP BY country, prov, city, ymd; ``` **Constraints for `hg_id_encoding_int4`:** - Mapping table must have exactly: one `text` PK column + one `serial` column - Input UID must not be NULL - Requires Hologres V4.1+ - Supports incremental refresh in Dynamic Tables ## Core RoaringBitmap Functions | Function | Description | Example | |----------|-------------|---------| | `RB_BUILD_AGG(int)` | Aggregate integers into a bitmap | `RB_BUILD_AGG(uid)` | | `RB_OR_AGG(roaringbitmap)` | Merge bitmaps (union / deduplicate) | `RB_OR_AGG(rb_uid)` | | `RB_AND_AGG(roaringbitmap)` | Intersect bitmaps (common users) | `RB_AND_AGG(rb_uid)` | | `RB_CARDINALITY(roaringbitmap)` | Count distinct elements in bitmap | `RB_CARDINALITY(rb)` | | `RB_OR(rb, rb)` | Union two bitmaps | `RB_OR(a, b)` | | `RB_AND(rb, rb)` | Intersect two bitmaps | `RB_AND(a, b)` | | `RB_ANDNOT(rb, rb)` | Difference (in A but not in B) | `RB_ANDNOT(a, b)` | | `RB_TO_ARRAY(roaringbitmap)` | Convert bitmap to integer array | `RB_TO_ARRAY(rb)` | | `RB_BUILD(int[])` | Build bitmap from integer array | `RB_BUILD(ARRAY[1,2,3])` | | `RB_CONTAINS(rb, int)` | Check if bitmap contains a value | `RB_CONTAINS(rb, 42)` | | `RB_IS_EMPTY(roaringbitmap)` | Check if bitmap is empty | `RB_IS_EMPTY(rb)` | For complete function reference, see [references/roaringbitmap-functions.md](references/roaringbitmap-functions.md). ## References | Document | Content | |----------|---------| | [roaringbitmap-functions.md](references/roaringbitmap-functions.md) | Complete RoaringBitmap function reference | | [dynamic-table-patterns.md](references/dynamic-table-patterns.md) | Dynamic Table configuration patterns for UV | | [advanced-scenarios.md](references/advanced-scenarios.md) | Advanced UV scenarios: retention, funnel, cross-platform | ## Best Practices 1. **Always use `LOGICAL PARTITION BY LIST (ymd)`** on both ODS and DWS tables for time-range partition pruning 2. **Set `auto_refresh_mode = 'incremental'`** to avoid recomputing all data on each refresh 3. **Configure [REDACTED_SECRET]** to limit auto-refresh to recent partitions only 4. **Manually refresh historical partitions** with `REFRESH DYNAMIC TABLE ... PARTITION(...) WITH (refresh_mode = 'full')` 5. **Use `RB_OR_AGG` for cross-day UV** — it merges daily bitmaps for exact deduplication over any date range 6. **For text UIDs, use `hg_id_encoding_int4`** (V4.1+) instead of manual mapping tables 7. **Set `distribution_key = 'uid'`** on ODS table for aggregation locality 8. **Install extension first** — `CREATE EXTENSION IF NOT EXISTS roaringbitmap` before any bitmap operations

path: VERSION
content: 0.2.0


path: package.json
content: {
  "name": "hologres-uv-compute",
  "version": "1.0.1",
  "description": "Hologres UV/PV computation using Dynamic Tables and RoaringBitmap for real-time deduplication at scale. Use for building incremental UV/PV pipelines, RoaringBitmap-based user deduplication, flexible time-range UV aggregation, and text-to-int UID encoding for bitmap compatibility. Triggers: \"hologres uv\", \"hologres pv\", \"roaringbitmap\", \"rb_build_agg\", \"rb_or_agg\", \"去重\", \"UV计算\", \"用户去重\", \"bitmap去重\", \"实时UV\", \"hg_id_encoding\"",
  "publishConfig": {
    "registry": "https://contextlab.alibaba-inc.com/skill"
  },
  "aoneKit": {
    "generated": true
  }
}


path: references/advanced-scenarios.md
content: # Advanced UV Scenarios

Advanced use cases beyond basic UV/PV: retention analysis, new/lost user tracking, multi-platform dedup, and funnel analysis using RoaringBitmap.

## Retention Analysis

Calculate day-N retention using bitmap intersection.

### Day-1 Retention

```sql
WITH day0 AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb WHERE ymd = '20251223'
),
day1 AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb WHERE ymd = '20251224'
)
SELECT
  RB_CARDINALITY(d0.rb) AS day0_uv,
  RB_CARDINALITY(RB_AND(d0.rb, d1.rb)) AS retained,
  ROUND(RB_CARDINALITY(RB_AND(d0.rb, d1.rb))::numeric
        / NULLIF(RB_CARDINALITY(d0.rb), 0) * 100, 2) AS retention_pct
FROM day0 d0, day1 d1;
```

### Multi-Day Retention (Day 1, 3, 7, 30)

```sql
WITH base AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb WHERE ymd = '20251201'
)
SELECT
  RB_CARDINALITY(b.rb) AS base_uv,
  RB_CARDINALITY(RB_AND(b.rb, (SELECT RB_OR_AGG(rb_uid) FROM dt_dws_app_rb WHERE ymd = '20251202'))) AS d1,
  RB_CARDINALITY(RB_AND(b.rb, (SELECT RB_OR_AGG(rb_uid) FROM dt_dws_app_rb WHERE ymd = '20251204'))) AS d3,
  RB_CARDINALITY(RB_AND(b.rb, (SELECT RB_OR_AGG(rb_uid) FROM dt_dws_app_rb WHERE ymd = '20251208'))) AS d7,
  RB_CARDINALITY(RB_AND(b.rb, (SELECT RB_OR_AGG(rb_uid) FROM dt_dws_app_rb WHERE ymd = '20251231'))) AS d30
FROM base b;
```

## New User & Lost User Analysis

### New Users (First Appearance)

Users who appeared in current period but not in previous period.

```sql
WITH current_period AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE ymd >= '20251224' AND ymd <= '20251230'
),
previous_period AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE ymd >= '20251217' AND ymd <= '20251223'
)
SELECT RB_CARDINALITY(RB_ANDNOT(c.rb, p.rb)) AS new_users
FROM current_period c, previous_period p;
```

### Lost Users (Churned)

Users who appeared in previous period but not in current period.

```sql
WITH current_period AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE ymd >= '20251224' AND ymd <= '20251230'
),
previous_period AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE ymd >= '20251217' AND ymd <= '20251223'
)
SELECT RB_CARDINALITY(RB_ANDNOT(p.rb, c.rb)) AS lost_users
FROM current_period c, previous_period p;
```

## Segment Overlap Analysis

Compare user overlap between dimensions (e.g., countries, channels).

```sql
WITH segment_cn AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE country = 'CN' AND ymd = '20251223'
),
segment_us AS (
  SELECT RB_OR_AGG(rb_uid) AS rb
  FROM dt_dws_app_rb
  WHERE country = 'US' AND ymd = '20251223'
)
SELECT
  RB_CARDINALITY(a.rb) AS cn_uv,
  RB_CARDINALITY(b.rb) AS us_uv,
  RB_CARDINALITY(RB_AND(a.rb, b.rb)) AS overlap,
  RB_CARDINALITY(RB_OR(a.rb, b.rb)) AS union_uv,
  ROUND(RB_CARDINALITY(RB_AND(a.rb, b.rb))::numeric
        / NULLIF(RB_CARDINALITY(RB_OR(a.rb, b.rb)), 0) * 100, 2)
    AS overlap_pct
FROM segment_cn a, segment_us b;
```

## Multi-Dimension UV in One Query

Calculate UV per dimension while also computing total UV across all dimensions.

```sql
-- Per-country UV + total UV
SELECT
  COALESCE(country, 'TOTAL') AS country,
  RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv,
  SUM(pv) AS pv
FROM dt_dws_app_rb
WHERE ymd >= '20251201' AND ymd <= '20251231'
GROUP BY ROLLUP(country)
ORDER BY uv DESC;
```

## Multiple Bitmap Columns

For more complex scenarios, store multiple bitmaps in one Dynamic Table.

```sql
-- Track both active users and paying users
CREATE DYNAMIC TABLE dt_dws_user_segments (
  ymd, country,
  rb_all_users, rb_paying_users,
  pv, pay_count
)
LOGICAL PARTITION BY LIST (ymd)
WITH (
  freshness = '5 minutes',
  auto_refresh_mode = 'incremental',
  auto_refresh_partition_active_time = '2 days',
  partition_key_time_format = 'YYYYMMDD'
)
AS
SELECT ymd, country,
       RB_BUILD_AGG(uid) AS rb_all_users,
       RB_BUILD_AGG(CASE WHEN is_paying THEN uid ELSE NULL END) AS rb_paying_users,
       COUNT(1) AS pv,
       SUM(CASE WHEN is_paying THEN 1 ELSE 0 END) AS pay_count
FROM ods_user_events
GROUP BY ymd, country;

-- Query: paying user conversion rate
SELECT country,
       RB_CARDINALITY(RB_OR_AGG(rb_all_users)) AS total_uv,
       RB_CARDINALITY(RB_OR_AGG(rb_paying_users)) AS paying_uv,
       ROUND(RB_CARDINALITY(RB_OR_AGG(rb_paying_users))::numeric
             / NULLIF(RB_CARDINALITY(RB_OR_AGG(rb_all_users)), 0) * 100, 2)
         AS conversion_pct
FROM dt_dws_user_segments
WHERE ymd >= '20251201' AND ymd <= '20251231'
GROUP BY country;
```

## Performance Tips

| Tip | Description |
|-----|-------------|
| Minimize GROUP BY dimensions | Fewer dimensions = fewer bitmap rows = faster merge |
| Filter before merge | Use WHERE to reduce partitions scanned before `RB_OR_AGG` |
| Use `LOGICAL PARTITION` | Enables partition pruning for time-range queries |
| Avoid `RB_TO_ARRAY` on large bitmaps | Converting millions of IDs to arrays is expensive |
| Pre-aggregate at coarser granularity | If only country-level UV is needed, don't include city in DWS |


path: references/dynamic-table-patterns.md
content: # Dynamic Table Configuration Patterns for UV

Configuration patterns and best practices for Dynamic Tables in UV/PV computation pipelines.

## Basic Configuration

### Minimal DWS Table

```sql
CREATE DYNAMIC TABLE dt_dws_app_rb (
  country, prov, city, rb_uid, pv, ymd
)
LOGICAL PARTITION BY LIST (ymd)
WITH (
  freshness = '5 minutes',
  auto_refresh_mode = 'incremental',
  auto_refresh_partition_active_time = '2 days',
  partition_key_time_format = 'YYYYMMDD'
)
AS
SELECT country, prov, city,
       RB_BUILD_AGG(uid) AS rb_uid,
       COUNT(1) AS pv,
       ymd
FROM ods_app_detail
GROUP BY country, prov, city, ymd;
```

### Key Parameter Reference

| Parameter | Description | Recommended |
|-----------|-------------|-------------|
| `freshness` | How fresh data should be | `5 minutes` for near-real-time, `1 hours` for batch |
| `auto_refresh_mode` | Refresh strategy | `incremental` — only new data |
| [REDACTED_SECRET] | Auto-refresh window | `2 days` — only refresh recent partitions |
| `partition_key_time_format` | Parse format for partition key | `YYYYMMDD`, `YYYY-MM-DD`, `YYYYMMDDHH24` |

## Freshness Tuning

Choose freshness based on business requirements:

| Scenario | Freshness | Trade-off |
|----------|-----------|-----------|
| Real-time dashboard | `1 minutes` | Higher compute cost |
| Near-real-time reporting | `5 minutes` | Balanced |
| Hourly batch | `1 hours` | Lower cost |
| Daily batch | `24 hours` | Minimal cost |

## Partition Active Time

Controls which partitions are auto-refreshed. Older partitions are NOT refreshed automatically.

```sql
-- Only auto-refresh last 2 days
auto_refresh_partition_active_time = '2 days'

-- Auto-refresh last 7 days
auto_refresh_partition_active_time = '7 days'
```

**Rule of thumb:** Set to cover the typical data correction window plus one day buffer.

## Historical Partition Refresh

Partitions outside `auto_refresh_partition_active_time` need manual refresh:

```sql
-- Refresh a single partition (full mode required for historical)
REFRESH DYNAMIC TABLE dt_dws_app_rb
  PARTITION(20251201)
  WITH (refresh_mode = 'full');

-- Refresh with OVERWRITE (replace existing data)
REFRESH OVERWRITE DYNAMIC TABLE dt_dws_app_rb
  PARTITION(20251201)
  WITH (refresh_mode = 'full');
```

## Storage Optimization

### Column Properties

```sql
CREATE DYNAMIC TABLE dt_dws_app_rb (...)
WITH (
  freshness = '5 minutes',
  auto_refresh_mode = 'incremental',
  auto_refresh_partition_active_time = '2 days',
  partition_key_time_format = 'YYYYMMDD',
  -- Storage optimization
  orientation = 'column',
  distribution_key = 'country',
  clustering_key = 'ymd',
  bitmap_columns = 'country,prov,city'
)
AS ...;
```

| Property | Recommendation | Reason |
|----------|---------------|--------|
| `orientation` | `column` | UV queries scan dimension columns |
| `distribution_key` | Primary GROUP BY dimension | Reduce shuffle |
| `clustering_key` | `ymd` | Optimize date-range filter |
| `bitmap_columns` | Dimension columns | Accelerate WHERE filters |

## Computing Resource Options

```sql
-- Use local compute (default)
computing_resource = 'local'

-- Use serverless compute (no impact on online queries)
computing_resource = 'serverless'

-- Use a specific warehouse
computing_resource = 'my_warehouse'
```

**Recommendation:** Use `serverless` for large DWS tables to isolate refresh load from query workload.

## Multi-Layer Pipeline

For complex scenarios, chain multiple Dynamic Tables:

```
ODS (detail) → DWD (cleaned) → DWS (bitmap agg) → Query
```

### DWD Layer (Optional Cleaning)

```sql
CREATE DYNAMIC TABLE dt_dwd_app_clean (...)
WITH (freshness = '3 minutes', auto_refresh_mode = 'incremental', ...)
AS
SELECT uid, country, prov, city, ymd
FROM ods_app_detail
WHERE uid IS NOT NULL AND country IS NOT NULL;
```

### DWS Layer (Bitmap Aggregation)

```sql
CREATE DYNAMIC TABLE dt_dws_app_rb (...)
WITH (freshness = '5 minutes', auto_refresh_mode = 'incremental', ...)
AS
SELECT country, prov, city,
       RB_BUILD_AGG(uid) AS rb_uid,
       COUNT(1) AS pv, ymd
FROM dt_dwd_app_clean
GROUP BY country, prov, city, ymd;
```

## Monitoring Refresh Status

```sql
-- Check Dynamic Table properties and refresh info
SELECT * FROM hologres.hg_dynamic_table_properties
WHERE table_name = 'dt_dws_app_rb';

-- Using CLI
-- hologres dt show dt_dws_app_rb
-- hologres dt list
```

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| Historical partition has no data | Not auto-refreshed | `REFRESH DYNAMIC TABLE ... PARTITION(...) WITH (refresh_mode = 'full')` |
| Refresh taking too long | Too many active partitions | Reduce `auto_refresh_partition_active_time` |
| Incremental refresh fails | Unsupported function used | Check [supported functions list](https://help.aliyun.com/zh/hologres/user-guide/supported-functions-for-incremental-refresh) |
| Data freshness not met | Compute resource insufficient | Switch to `serverless` or increase warehouse capacity |


path: references/roaringbitmap-functions.md
content: # RoaringBitmap Function Reference

Complete function reference for RoaringBitmap operations in Hologres.

## Prerequisites

```sql
CREATE EXTENSION IF NOT EXISTS roaringbitmap;
```

## Aggregate Functions

These functions aggregate multiple rows into a single bitmap.

### RB_BUILD_AGG

Build a bitmap from an integer column across rows.

```sql
-- Syntax
RB_BUILD_AGG(integer_column) → roaringbitmap

-- Example: aggregate all UIDs into one bitmap
SELECT RB_BUILD_AGG(uid) AS user_bitmap
FROM ods_app_detail
WHERE ymd = '20251223';
```

**Notes:**
- Input must be `int4` (integer). For `bigint`, use `RB_BUILD_AGG_INT8`.
- NULL values are ignored.
- Supported in Dynamic Table incremental refresh.

### RB_OR_AGG

Merge (union) multiple bitmaps. Result contains all distinct elements from all input bitmaps.

```sql
-- Syntax
RB_OR_AGG(roaringbitmap_column) → roaringbitmap

-- Example: merge daily bitmaps for monthly UV
SELECT RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS monthly_uv
FROM dt_dws_app_rb
WHERE ymd >= '20251201' AND ymd <= '20251231';
```

**Use case:** Cross-day/week/month UV deduplication.

### RB_AND_AGG

Intersect multiple bitmaps. Result contains only elements present in ALL input bitmaps.

```sql
-- Syntax
RB_AND_AGG(roaringbitmap_column) → roaringbitmap

-- Example: users who visited EVERY day in a week
SELECT RB_CARDINALITY(RB_AND_AGG(rb_uid)) AS daily_active_users
FROM dt_dws_app_rb
WHERE ymd >= '20251223' AND ymd <= '20251229';
```

**Use case:** Retention analysis — find users present across all periods.

## Scalar Functions (Two-Bitmap Operations)

### RB_OR

Union two bitmaps.

```sql
-- Syntax
RB_OR(roaringbitmap, roaringbitmap) → roaringbitmap

-- Example
SELECT RB_CARDINALITY(RB_OR(bitmap_day1, bitmap_day2)) AS two_day_uv;
```

### RB_AND

Intersect two bitmaps.

```sql
-- Syntax
RB_AND(roaringbitmap, roaringbitmap) → roaringbitmap

-- Example: users in both groups
SELECT RB_CARDINALITY(RB_AND(group_a_bitmap, group_b_bitmap)) AS overlap;
```

### RB_ANDNOT

Difference — elements in the first bitmap but NOT in the second.

```sql
-- Syntax
RB_ANDNOT(roaringbitmap, roaringbitmap) → roaringbitmap

-- Example: new users today (in today but not in yesterday)
SELECT RB_CARDINALITY(RB_ANDNOT(today_bitmap, yesterday_bitmap)) AS new_users;
```

### RB_XOR

Symmetric difference — elements in either bitmap but not both.

```sql
-- Syntax
RB_XOR(roaringbitmap, roaringbitmap) → roaringbitmap
```

## Cardinality & Inspection

### RB_CARDINALITY

Count distinct elements in a bitmap.

```sql
-- Syntax
RB_CARDINALITY(roaringbitmap) → bigint

-- Example
SELECT RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv
FROM dt_dws_app_rb
WHERE ymd = '20251223';
```

### RB_IS_EMPTY

Check if a bitmap has no elements.

```sql
-- Syntax
RB_IS_EMPTY(roaringbitmap) → boolean
```

### RB_CONTAINS

Check if a bitmap contains a specific integer value.

```sql
-- Syntax
RB_CONTAINS(roaringbitmap, integer) → boolean

-- Example: check if user 12345 is in the bitmap
SELECT RB_CONTAINS(rb_uid, 12345) FROM dt_dws_app_rb WHERE ymd = '20251223';
```

### RB_EQUALS

Check if two bitmaps are identical.

```sql
-- Syntax
RB_EQUALS(roaringbitmap, roaringbitmap) → boolean
```

### RB_INTERSECT

Check if two bitmaps share any element.

```sql
-- Syntax
RB_INTERSECT(roaringbitmap, roaringbitmap) → boolean
```

## Construction & Conversion

### RB_BUILD

Build a bitmap from an integer array.

```sql
-- Syntax
RB_BUILD(integer[]) → roaringbitmap

-- Example
SELECT RB_BUILD(ARRAY[1, 2, 3, 100, 200]);
```

### RB_TO_ARRAY

Convert a bitmap back to an integer array.

```sql
-- Syntax
RB_TO_ARRAY(roaringbitmap) → integer[]

-- Example: get all user IDs from a bitmap
SELECT RB_TO_ARRAY(rb_uid) FROM dt_dws_app_rb LIMIT 1;
```

### RB_BUILD_AGG_INT8

Build a bitmap from a `bigint` column (64-bit input). Internally splits into high/low 32-bit parts.

```sql
-- Syntax
RB_BUILD_AGG_INT8(bigint_column) → roaringbitmap
```

## UID Encoding Functions

For text-type UIDs that need integer mapping before bitmap operations.

### hg_id_encoding_int4 (V4.1+)

Auto-map text UIDs to 32-bit integers using a mapping table.

```sql
-- Syntax
hg_id_encoding_int4(text_uid, 'mapping_table_name') → integer

-- Prerequisites: mapping table with text PK + serial column
CREATE TABLE uid_mapping (
  uid text NOT NULL PRIMARY KEY,
  uid_int32 serial
);

-- Usage in Dynamic Table
SELECT RB_BUILD_AGG(hg_id_encoding_int4(device_id, 'uid_mapping')) AS uid_rb
FROM ods_events
GROUP BY dimension;
```

**Constraints:**
- Mapping table: exactly one `text` PK + one `serial` column
- Input must not be NULL
- Requires Hologres V4.1+
- Supported in incremental Dynamic Table refresh

### hg_id_encoding_int8

Same as `hg_id_encoding_int4` but maps to `bigint` (64-bit).

## Common Patterns

### UV for Flexible Date Range

```sql
-- Works for any date range: day, week, month, quarter, year
SELECT RB_CARDINALITY(RB_OR_AGG(rb_uid)) AS uv
FROM dt_dws_app_rb
WHERE ymd >= '20251201' AND ymd <= '20251231';
```

### New Users (Appeared Today, Not Yesterday)

```sql
WITH today AS (
  SELECT RB_OR_AGG(rb_uid) AS rb FROM dt_dws_app_rb WHERE ymd = '20251224'
),
yesterday AS (
  SELECT RB_OR_AGG(rb_uid) AS rb FROM dt_dws_app_rb WHERE ymd = '20251223'
)
SELECT RB_CARDINALITY(RB_ANDNOT(t.rb, y.rb)) AS new_users
FROM today t, yesterday y;
```

### User Overlap Between Two Segments

```sql
WITH segment_a AS (
  SELECT RB_OR_AGG(rb_uid) AS rb FROM dt_dws_app_rb
  WHERE country = 'CN' AND ymd = '20251223'
),
segment_b AS (
  SELECT RB_OR_AGG(rb_uid) AS rb FROM dt_dws_app_rb
  WHERE country = 'US' AND ymd = '20251223'
)
SELECT RB_CARDINALITY(RB_AND(a.rb, b.rb)) AS overlap_uv,
       RB_CARDINALITY(RB_OR(a.rb, b.rb)) AS total_uv
FROM segment_a a, segment_b b;
```


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)