title_wip

This article explains how to recover data when using replicated tables in atomic databases in ClickHouse and disks/storage on one of the replica is lost/currupted.

  
  

Note

This guide assumes that the `<path>` parameter in your config.xml file is set to:

```
<path>/var/lib/clickhouse/</path>
```

If you have configured a different data path, replace all instances of `/var/lib/clickhouse` in the below commands with the actual value of your `<path>` setting.

Copy access configuration from the healthy replica

## Copy access configuration from the healthy replica[​](#copy-access-config "Direct link to Copy access configuration from the healthy replica")

Copy the contents of the `access` folder which contains local users from the healthy replica:

```
/var/lib/clickhouse/access
```

Back up the metadata folder from the healthy replica

## Back up the metadata folder from the healthy replica[​](#back-up-the-metadata-folder-from-the-healthy-replica "Direct link to Back up the metadata folder from the healthy replica")

1. Navigate to the ClickHouse data directory:

```
cd /var/lib/clickhouse
```

2. Create a backup of the metadata folder (including symbolic links): The metadata directory contains DDLs for databases and tables.
   The database directory has symlinks to `/var/lib/clickhouse/store/..` which contains all the table DDLs.

```
{ find metadata -type f; find metadata -type l; find metadata -type l | xargs readlink -f; } | tar -cPf backup.tar --files-from=-
```

Note

This command ensures that both the **metadata files**, and the symlink architecture are preserved in the backup.

Restore the metadata on the faulty replica

## Restore the metadata on the faulty replica[​](#restore-the-metadata-on-the-faulty-replica "Direct link to Restore the metadata on the faulty replica")

1. Copy the generated `backup.tar` file to the faulty replica.
2. Extract it to the ClickHouse data directory:

```
cd /var/lib/clickhouse/
tar -xvPf backup.tar
```

Create the force restore flag

## Create the force restore flag[​](#create-force-restore-flag "Direct link to Create the force restore flag")

To trigger automatic data synchronization from other replicas, create the following flag:

```
sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data
```

Restart the faulty replica

## Restart the faulty replica[​](#restart-faulty-replica "Direct link to Restart the faulty replica")

1. Restart the ClickHouse server on the faulty node.
2. Check the server logs, you should observe parts being downloaded from the healthy replicas:

```
2025.11.02 00:00:04.047097 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Downloading files 23
2025.11.02 00:00:04.055542 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Download of part 202511_0_0_0 onto disk disk2 finished.
2025.11.02 00:00:04.101888 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2025_0_0_1 onto disk default.
2025.11.02 00:00:04.102005 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
2025.11.02 00:00:04.102210 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2022_0_0_1 onto disk disk1.
2025.11.02 00:00:04.102247 [ 688 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2021_0_0_1 onto disk disk2.
2025.11.02 00:00:04.102331 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
```

Nov 19, 2025 · 2 min read