How to recover from a corrupt Keeper snapshot

Article describing how to recover from a corrupt Keeper snapshot: how the problem manifests, what a snapshot is and where to find it and possible recovery strategies.

  
  

Corrupt or bad ClickHouse Keeper snapshots can cause significant system instability, such as metadata inconsistencies, read-only states for tables, resource exhaustion, or failed backups. This article covers:

- [What snapshots are and where to find them](#overview)
- [How the problem manifests](#symptoms)
- [Possible strategies for recovery](#recovery-strategies) and what each of them means

## Overview of Keeper snapshots[​](#overview "Direct link to Overview of Keeper snapshots")

### What is a snapshot?[​](#what-is-snapshot "Direct link to What is a snapshot?")

A snapshot is a serialized state of Keeper's internal data (such as metadata about clusters, table coordination paths, and configurations) at a specific point in time. Snapshots are vital for resynchronizing Keeper nodes within a cluster, recovering metadata during failures, and start-up or restart processes that rely on a known-good Keeper state.

### Where can I find snapshots?[​](#where-to-find-snapshots "Direct link to Where can I find snapshots?")

Snapshots are stored as files on the local filesystem of Keeper nodes. By default, they are stored at `/var/lib/clickhouse/coordination/snapshots/` or by the custom path specified by `snapshot_storage_path` in your `keeper_server.xml` file. Snapshots are named incrementally (e.g., snapshot.23), with newer ones having higher numbers.

For multi-node clusters, each Keeper node has its own snapshot directory.

Note

Consistency within snapshots across nodes is critical for recovery.

## Key symptoms and manifestations of corrupt Keeper snapshots[​](#symptoms "Direct link to Key symptoms and manifestations of corrupt Keeper snapshots")

The table below details some common symptoms and manifestations of corrupt Keeper snapshots:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **Category** **Issue Type** **What to look for**|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Operational Issues** Read-Only Mode Tables unexpectedly switch to read-only mode|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Query Failures Persistent query failures with `Coordination::Exception` errors| **Metadata Corruption** Outdated Metadata Dropped tables not reflected; operation failures due to stale metadata|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Resource Overload** System Resource Exhaustion Keeper nodes consume excessive CPU, memory, or disk space; potential downtime|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Disk Full Disk full during snapshot creation|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Backup & Restore** Backup Failures Backups fail due to missing or inconsistent Keeper metadata|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Snapshot Creation/Transfer** Keeper Crash Keeper crash mid-snapshot (look for "SEGFAULT" errors)|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Snapshot Transfer Corruption Corruption during snapshot transfer between replicas|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | Race Condition Race condition during log compaction - background commit thread accessing deleted logs|  |  |  | | --- | --- | --- | | Network Synchronization Network issues preventing snapshot sync from leader to followers | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

**Log Indicators:**

Before diagnosing snapshot corruption, check **Keeper logs** for specific error patterns:

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| **Log Type** **What to Look For**|  |  |  |  | | --- | --- | --- | --- | | **Snapshot corruption errors** • `Aborting because of failure to load from latest snapshot with index` • `Failure to load from latest snapshot with index {}: {}. Manual intervention is necessary for recovery` • `Failed to preprocess stored log at index {}, aborting to avoid inconsistent state` • Snapshot serialization/loading failures during startup| **Other Keeper issues** • `Coordination::Exception` • `Zookeeper::Session Timeout` • Synchronization or election issues • Log compaction race conditions | | | | | |

## Recovering from corrupt Keeper snapshots[​](#recovery-strategies "Direct link to Recovering from corrupt Keeper snapshots")

Before touching any files, always:

1. Stop all Keeper nodes to prevent further corruption
2. Backup everything by copying the entire coordination directory to a safe location
3. Verify cluster quorum to ensure at least one node has good data

---

### 1. Restore from an existing backup[​](#1-restore-from-an-existing-backup "Direct link to 1. Restore from an existing backup")

You should follow this process if:

- The Keeper metadata or snapshot corruption makes current data unsalvageable.
- A backup exists with a known-good Keeper state.

Follow the steps below to restore an existing backup:

1. Locate and validate the newest backup for metadata consistency.
2. Shut down the ClickHouse and Keeper services.
3. Replace the faulty snapshots and logs with those from the backup directory.
4. Restart the Keeper cluster and validate metadata synchronization.

Backup regularly

If backups are outdated, you may incur a loss of recent metadata changes. For this reason, we recommend backing up regularly.

---

### 2. Rollback to an older snapshot[​](#2-rollback-to-an-older-snapshot "Direct link to 2. Rollback to an older snapshot")

You should follow this process when:

- Recent snapshots are corrupt, but older ones remain usable.
- Incremental logs are intact for consistent recovery.

Follow the steps below to roll back to an older snapshot:

1. Identify and select a valid older snapshot (e.g., snapshot.19) from the Keeper directory.
2. Remove newer snapshots and logs.
3. Restart Keeper so it replays logs to rebuild the metadata state.

Metadata desynchronization risk

There is a risk of metadata desynchronization if snapshots and logs are missing or incomplete.

---

### 3. Restore metadata using SYSTEM RESTORE REPLICA[​](#3-restore-metadata-using-system-restore-replica "Direct link to 3. Restore metadata using SYSTEM RESTORE REPLICA")

You should follow this process when:

- Keeper metadata is lost or corrupted but table data still exists on disk
- Tables have switched to read-only mode due to missing ZooKeeper/Keeper metadata
- You need to recreate metadata in Keeper based on locally available data parts

Follow the steps below to restore metadata:

1. Verify that table data exists locally in your clickHouse-server data path, set by `<path>` in your config. (`/var/lib/clickhouse/data/` by default)
2. For each affected table, execute:

```
SYSTEM RESTART REPLICA [db.]table_name;
SYSTEM RESTORE REPLICA [db.]table_name;
```

3. For database-level recovery (if using Replicated database engine):

```
SYSTEM RESTORE DATABASE REPLICA db_name;
```

4. Wait for synchronization to complete:

```
SYSTEM SYNC REPLICA [db.]table_name;
```