How to recover from a corrupt Keeper snapshot

You should follow this process when:

- Keeper metadata is lost or corrupted but table data still exists on disk
- Tables have switched to read-only mode due to missing ZooKeeper/Keeper metadata
- You need to recreate metadata in Keeper based on locally available data parts

Follow the steps below to restore metadata:

1. Verify that table data exists locally in your clickHouse-server data path, set by `<path>` in your config. (`/var/lib/clickhouse/data/` by default)
2. For each affected table, execute:

```
SYSTEM RESTART REPLICA [db.]table_name;
SYSTEM RESTORE REPLICA [db.]table_name;
```

3. For database-level recovery (if using Replicated database engine):

```
SYSTEM RESTORE DATABASE REPLICA db_name;
```

4. Wait for synchronization to complete:

```
SYSTEM SYNC REPLICA [db.]table_name;
```

5. Verify recovery by checking `system.replicas` for `is_readonly = 0` and monitoring `system.detached_parts`

How it works

`SYSTEM RESTORE REPLICA` detaches all existing parts, recreates metadata in Keeper (as if it's a new empty table), then reattaches all parts. This avoids re-downloading data over the network.

Prerequisites

This only works if local data parts are intact. If data is also corrupted, use strategy #5 (rebuild cluster) instead.

---

### 4. Drop and recreate replica metadata in Keeper[​](#4-drop-and-recreate-replica-metadata-in-keeper "Direct link to 4. Drop and recreate replica metadata in Keeper")

You should follow this process when:

- The error occurs on a single replica of the cluster and has corrupt or inconsistent metadata in Keeper
- You encounter errors like "Part XXXXX intersects previous part YYYYY"
- You need to completely reset a replica's Keeper metadata while preserving local data

Follow the steps below to drop and recreate metadata:

1. On the affected replica, detach the table:

```
DETACH TABLE [db.]table_name;
```

2. Remove the replica's metadata from Keeper (execute on any replica):

```
SYSTEM DROP REPLICA 'replica_name' FROM ZKPATH '/clickhouse/tables/{shard}/table_name';
```

To find the correct ZooKeeper path:

```
SELECT zookeeper_path, replica_name FROM system.replicas WHERE table = 'table_name';
```

3. Reattach the table (it will be in read-only mode):

```
ATTACH TABLE [db.]table_name;
```

4. Restore the replica metadata:

```
SYSTEM RESTORE REPLICA [db.]table_name;
```

5. Synchronize with other replicas:

```
SYSTEM SYNC REPLICA [db.]table_name;
```

6. Check `system.detached_parts` on all replicas after recovery

Execute on all affected replicas

If the corruption affects multiple replicas, repeat these steps on each one sequentially.

For entire database

If using a Replicated database, you can use `SYSTEM DROP REPLICA ... FROM DATABASE db_name` instead.

**Alternative: Using force\_restore\_data flag**

For automatic recovery of all replicated tables at server startup:

1. Stop ClickHouse server
2. Create the recovery flag:

```
sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data
```

3. Start ClickHouse server
4. The server will automatically delete the flag and restore all replicated tables
5. Monitor logs for recovery progress

This approach is useful when multiple tables need recovery simultaneously.

---

### 5. Rebuild Keeper cluster[​](#5-rebuild-keeper-cluster "Direct link to 5. Rebuild Keeper cluster")

You should follow this process when:

- No valid snapshots, logs, or backups are available for recovery.
- You need to recreate the entire Keeper cluster and its metadata.

Follow the steps below to rebuild the Keeper cluster:

1. Fully stop the ClickHouse and Keeper clusters.
2. Reset each Keeper node by cleaning the snapshot and log directories.
3. Initialize one Keeper node as the leader and add other nodes incrementally.
4. Re-import metadata if available from external records.

Time-intensive process

This process is time-intensive and carries a risk of prolonged outage. Total data reconstruction is required.

Jan 29, 2025 · 7 min read