Migrating data to ClickStack from Elastic

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/use-cases/observability/clickstack/migration/elastic/migrating-data.md)

## Parallel operation strategy[​](#parallel-operation-strategy "Direct link to Parallel operation strategy")

When migrating from Elastic to ClickStack for observability use cases, we recommend a **parallel operation** approach rather than attempting to migrate historical data. This strategy offers several advantages:

1. **Minimal risk**: by running both systems concurrently, you maintain access to existing data and dashboards while validating ClickStack and familiarizing your users with the new system.
2. **Natural data expiration**: most observability data has a limited retention period (typically 30 days or less), allowing for a natural transition as data expires from Elastic.
3. **Simplified migration**: no need for complex data transfer tools or processes to move historical data between systems.

  

Migrating data

We demonstrate an approach for migrating essential data from Elasticsearch to ClickHouse in the section ["Migrating data"](#migrating-data). This shouldn't be used for larger datasets as it is rarely performant - limited by the ability for Elasticsearch to export efficiently, with only JSON format supported.

### Implementation steps[​](#implementation-steps "Direct link to Implementation steps")

Configure dual ingestion

#### Configure dual ingestion[​](#configure-dual-ingestion "Direct link to Configure dual ingestion")

Set up your data collection pipeline to send data to both Elastic and ClickStack simultaneously.

How this is achieved depends on your current agents for collection - see ["Migrating Agents"](/docs/use-cases/observability/clickstack/migration/elastic/migrating-agents).

Adjust retention periods

#### Adjust retention periods[​](#adjust-retention-period "Direct link to Adjust retention periods")

Configure Elastic's TTL settings to match your desired retention period. Set up the ClickStack [TTL](/docs/use-cases/observability/clickstack/production#configure-ttl) to maintain data for the same duration.

Validate and compare

#### Validate and compare[​](#validate-and-compare "Direct link to Validate and compare")

- Run queries against both systems to ensure data consistency
- Compare query performance and results
- Migrate dashboards and alerts to ClickStack. This is currently a manual process.
- Verify that all critical dashboards and alerts work as expected in ClickStack

Gradual transition

#### Gradual transition[​](#graudal-transition "Direct link to Gradual transition")

- As data naturally expires from Elastic, you will increasingly rely on ClickStack
- Once confidence in ClickStack is established, you can begin redirecting queries and dashboards

### Long-term retention[​](#long-term-retention "Direct link to Long-term retention")

For organizations requiring longer retention periods:

- Continue running both systems in parallel until all data has expired from Elastic
- ClickStack [tiered storage](/docs/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes) capabilities can help manage long-term data efficiently.
- Consider using [materialized views](/docs/materialized-view/incremental-materialized-view) to maintain aggregated or filtered historical data while allowing raw data to expire.

### Migration timeline[​](#migration-timeline "Direct link to Migration timeline")

The migration timeline will depend on your data retention requirements:

- **30-day retention**: Migration can be completed within a month.
- **Longer retention**: Continue parallel operation until data expires from Elastic.
- **Historical data**: If absolutely necessary, consider using [Migrating data](#migrating-data) to import specific historical data.

## Migrating settings[​](#migration-settings "Direct link to Migrating settings")

When migrating from Elastic to ClickStack, your indexing and storage settings will need to be adapted to fit ClickHouse's architecture. While Elasticsearch relies on horizontal scaling and sharding for performance and fault tolerance and thus has multiple shards by default, ClickHouse is optimized for vertical scaling and typically performs best with fewer shards.

### Recommended settings[​](#recommended-settings "Direct link to Recommended settings")

We recommend starting with a **single shard** and scaling vertically. This configuration is suitable for most observability workloads and simplifies both management and query performance tuning.

- **[ClickHouse Cloud](https://clickhouse.com/cloud)**: Uses a single-shard, multi-replica architecture by default. Storage and compute scale independently, making it ideal for observability use cases with unpredictable ingest patterns and read-heavy workloads.
- **ClickHouse OSS**: In self-managed deployments, we recommend:
  - Starting with a single shard
  - Scaling vertically with additional CPU and RAM
  - Using [tiered storage](/docs/observability/managing-data#storage-tiers) to extend local disk with S3-compatible object storage
  - Using [`ReplicatedMergeTree`](/docs/engines/table-engines/mergetree-family/replication) if high availability is required
  - For fault tolerance, [1 replica of your shard](/docs/engines/table-engines/mergetree-family/replication) is typically sufficient in Observability workloads.

### When to shard[​](#when-to-shard "Direct link to When to shard")

Sharding may be necessary if:

- Your ingest rate exceeds the capacity of a single node (typically >500K rows/sec)
- You need tenant isolation or regional data separation
- Your total dataset is too large for a single server, even with object storage

If you do need to shard, refer to [Horizontal scaling](/docs/architecture/horizontal-scaling) for guidance on shard keys and distributed table setup.

### Retention and TTL[​](#retention-and-ttl "Direct link to Retention and TTL")

ClickHouse uses [TTL clauses](/docs/use-cases/observability/clickstack/production#configure-ttl) on MergeTree tables to manage data expiration. TTL policies can:

- Automatically delete expired data
- Move older data to cold object storage
- Retain only recent, frequently queried logs on fast disk

We recommend aligning your ClickHouse TTL configuration with your existing Elastic retention policies to maintain a consistent data lifecycle during the migration. For examples, see [ClickStack production TTL setup](/docs/use-cases/observability/clickstack/production#configure-ttl).

## Migrating data[​](#migrating-data "Direct link to Migrating data")

While we recommend parallel operation for most observability data, there are specific cases where direct data migration from Elasticsearch to ClickHouse may be necessary:

- Small lookup tables used for data enrichment (e.g., user mappings, service catalogs)
- Business data stored in Elasticsearch that needs to be correlated with observability data, with ClickHouse's SQL capabilities and Business Intelligence integrations making it easier to maintain and query the data compared to Elasticsearch's more limited query options.
- Configuration data that needs to be preserved across the migration

This approach is only viable for datasets under 10 million rows, as Elasticsearch's export capabilities are limited to JSON over HTTP and don't scale well for larger datasets.

The following steps allow the migration of a single Elasticsearch index from ClickHouse.

Migrate schema

### Migrate schema[​](#migrate-scheme "Direct link to Migrate schema")

Create a table in ClickHouse for the index being migrated from Elasticsearch. You can map [Elasticsearch types to their ClickHouse](/docs/use-cases/observability/clickstack/migration/elastic/types) equivalent. Alternatively, you can simply rely on the JSON data type in ClickHouse, which will dynamically create columns of the appropriate type as data is inserted.

Consider the following Elasticsearch mapping for an index containing `syslog` data:Elasticsearch mapping