Repository: camunda/camunda
Pull request #29309: [POC]: Add Standalone backups application
Author: login: ChrisKujawa
Labels: name: component/zeebe
description: Related to the Zeebe component/team
name: component/operate
description: Related to the Operate component/team
name: component/tasklist
description: Related to the Tasklist component/team
name: component/optimize
description: Related to Optimize component/team
name: backport stable/8.7
description: Backport a pull request to stable/8.7
name: version:8.7.0
description: Marks an issue as being completely or in parts released in 8.7.0
Created: 2025-03-06T14:39:57Z | Merged: 2025-03-10T17:28:39Z

## Description

We have the use case of creating backups (ES index snapshots) separate of our Camunda Platform, as taking snapshot in ES/OS requires separate and even higher permissions. 

There is the need of separating concerns here and to not have Camunda 8 application running all the time with such high permissions. 

As a POC we were building a SpringBoot application that:

- Accepts a backupId via command line argument
- That can be configured with similar properties we had before for the [StandaloneSchmemaManager](https://docs.camunda.io/docs/self-managed/concepts/elasticsearch-without-cluster-privileges/#11-configure-schema-manager)
    - Additionally, we would need to configure
        - `camunda.operate.backup.repositoryName:`
        - `camunda.tasklist.backup.repositoryName:`
- Bootstraps all necessary dependencies
- Schedules Operate and Tasklist indices snapshots on ES snapshot store
- Observes the process of the snapshots, until it is completed or in a failed state
- The application can be executed as: `SPRING_CONFIG_ADDITIONALLOCATION=backup-manager.yml ./dist/target/camunda-zeebe/bin/backup <backupID>`
  - Similar to what we have for the StandaloneSchemaManager

## Execution / Manual verification

1. Setup ES container: `$ podman run -p 9200:9200 -e "xpack.security.enabled=false" -e  "discovery.type=single-node" -e "action.destructive_requires_name=false" -e "indices.lifecycle.poll_interval=1s" -e "path.repo=~/" docker.io/library/elasticsearch:8.17.2
 `
   Be aware that `"path.repo=~/"` is necessary to setup the snapshot repository in ES
2. Create a snapshot repository via ElasticVue - in browser

![2025-03-06_13-47_1](https://github.com/user-attachments/assets/83a29f30-391f-45d5-bca0-559b93b5870b)
![2025-03-06_13-47](https://github.com/user-attachments/assets/f7130642-b528-4f21-aa7c-8ed0c9359a0a)

3. Run StandaloneCamunda to set up indices and some data

![2025-03-06_10-32](https://github.com/user-attachments/assets/998f8505-2adb-470b-a9ad-aad36dc28675)

4. Run the standalone backup manager
![2025-03-06_14-44_1](https://github.com/user-attachments/assets/65cf082e-9f66-4a48-a64a-9d0ea585720e)
![2025-03-06_14-44](https://github.com/user-attachments/assets/def982f8-8300-49af-953a-77d10c5a8e2f)





Diff:
diff --git a/backup-manager.yml b/backup-manager.yml
new file mode 100644
index 000000000000..e26c54797f81
--- /dev/null
+++ b/backup-manager.yml
@@ -0,0 +1,17 @@
+camunda:
+  operate:
+    backup:
+      repositoryName: "els-test"
+    elasticsearch:
+      # Example assuming an existing user called 'camunda-admin' who has 'superuser' privileges
+      username: camunda-admin
+      password: camunda123
+      healthCheckEnabled: false
+  tasklist:
+    backup:
+      repositoryName: "els-test"
+    elasticsearch:
+      # Example assuming an existing user called 'camunda-admin' who has 'superuser' privileges
+      username: camunda-admin
+      password: camunda123
+      healthCheckEnabled: false
diff --git a/dist/pom.xml b/dist/pom.xml
index 6698051c02d7..e33f290dd3ad 100644
--- a/dist/pom.xml
+++ b/dist/pom.xml
@@ -265,11 +265,6 @@
       <artifactId>log4j-core</artifactId>
     </dependency>
 
-    <dependency>
-      <groupId>org.springframework.boot</groupId>
-      <artifactId>spring-boot</artifactId>
-    </dependency>
-
     <dependency>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-autoconfigure</artifactId>
@@ -550,6 +545,10 @@
       <artifactId>testcontainers</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.springframework.boot</groupId>
+      <artifactId>spring-boot</artifactId>
+    </dependency>
 
   </dependencies>
 
@@ -596,6 +595,10 @@
               <id>schema</id>
               <mainClass>io.camunda.application.StandaloneSchemaManager</mainClass>
             </program>
+            <program>
+              <id>backup-webapps</id>
+              <mainClass>io.camunda.application.StandaloneBackupManager</mainClass>
+            </program>
             <program>
               <id>tasklist</id>
               <mainClass>io.camunda.application.StandaloneTasklist</mainClass>
diff --git a/dist/src/main/java/io/camunda/application/StandaloneBackupManager.java b/dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
new file mode 100644
index 000000000000..a65ab534a716
--- /dev/null
+++ b/dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
@@ -0,0 +1,235 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.application;
+
+import io.camunda.application.StandaloneSchemaManager.SchemaManagerConfiguration.BrokerBasedProperties;
+import io.camunda.operate.webapp.backup.BackupService;
+import io.camunda.tasklist.webapp.es.backup.BackupManager;
+import io.camunda.tasklist.webapp.management.dto.BackupStateDto;
+import io.camunda.tasklist.webapp.management.dto.TakeBackupRequestDto;
+import java.util.Arrays;
+import java.util.EnumSet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.ExitCodeExceptionMapper;
+import org.springframework.boot.SpringBootConfiguration;
+import org.springframework.boot.WebApplicationType;
+import org.springframework.boot.context.properties.ConfigurationPropertiesScan;
+import org.springframework.boot.context.properties.EnableConfigurationProperties;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.FullyQualifiedAnnotationBeanNameGenerator;
+
+/**
+ * Backup Operate and Tasklist indices for ElasticSearch by running this standalone application.
+ *
+ * <p>Example properties:
+ *
+ * <pre>
+ * camunda.operate.elasticsearch.indexPrefix=operate
+ * camunda.operate.elasticsearch.clusterName=elasticsearch
+ * camunda.operate.elasticsearch.url=https://localhost:9200
+ * camunda.operate.elasticsearch.ssl.selfSigned=true
+ * camunda.operate.elasticsearch.ssl.verifyHostname=false
+ * camunda.operate.elasticsearch.ssl.certificatePath=C:/.../config/certs/http_ca.crt
+ * camunda.operate.elasticsearch.username=camunda-admin
+ * camunda.operate.elasticsearch.password=camunda123
+ *
+ * camunda.operate.backup.repositoryName=els-test
+ *
+ * camunda.tasklist.elasticsearch.indexPrefix=tasklist
+ * camunda.tasklist.elasticsearch.clusterName=elasticsearch
+ * camunda.tasklist.elasticsearch.url=https://localhost:9200
+ * camunda.tasklist.elasticsearch.ssl.selfSigned=true
+ * camunda.tasklist.elasticsearch.ssl.verifyHostname=false
+ * camunda.tasklist.elasticsearch.ssl.certificatePath=C:/.../config/certs/http_ca.crt
+ * camunda.tasklist.elasticsearch.username=camunda-admin
+ * camunda.tasklist.elasticsearch.password=camunda123
+ *
+ * camunda.tasklist.backup.repositoryName=els-test
+ *
+ * </pre>
+ *
+ * All of those properties can also be handed over via environment variables, e.g.
+ * `CAMUNDA_OPERATE_ELASTICSEARCH_INDEXPREFIX`
+ */
+public class StandaloneBackupManager implements CommandLineRunner {
+
+  private static final Logger LOG = LoggerFactory.getLogger(StandaloneBackupManager.class);
+  private final BackupManager tasklistBackupManager;
+  private final BackupService operateBackupManager;
+
+  public StandaloneBackupManager(
+      final BackupManager tasklistBackupManager, final BackupService operateBackupManager) {
+    this.tasklistBackupManager = tasklistBackupManager;
+    this.operateBackupManager = operateBackupManager;
+  }
+
+  public static void main(final String[] args) throws Exception {
+    // To ensure that debug logging performed using java.util.logging is routed into Log4j 2
+    MainSupport.putSystemPropertyIfAbsent(
+        "java.util.logging.manager", "org.apache.logging.log4j.jul.LogManager");
+    // Workaround for https://github.com/spring-projects/spring-boot/issues/26627
+    MainSupport.putSystemPropertyIfAbsent(
+        "spring.config.location",
+        "optional:classpath:/,optional:classpath:/config/,optional:file:./,optional:file:./config/");
+
+    // show banner
+    MainSupport.putSystemPropertyIfAbsent(
+        "spring.banner.location", "classpath:/assets/camunda_banner.txt");
+
+    LOG.info("Creating a backup for Operate and Tasklist elasticsearch indices ...");
+
+    MainSupport.createDefaultApplicationBuilder()
+        .web(WebApplicationType.NONE)
+        .logStartupInfo(true)
+        .sources(
+            BackupManagerConfiguration.class,
+            StandaloneBackupManager.class,
+            io.camunda.tasklist.JacksonConfig.class,
+            io.camunda.operate.JacksonConfig.class)
+        .addCommandLineProperties(true)
+        .run(args);
+
+    // Explicit exit needed because there are daemon threads (at least from the ES client) that are
+    // blocking shutdown.
+    System.exit(0);
+  }
+
+  @Override
+  public void run(final String... args) throws Exception {
+    if (args.length != 1) {
+      throw new IllegalArgumentException(
+          String.format("Expected one argument, the backup ID, but got: %s", Arrays.asList(args)));
+    }
+
+    final long backupId;
+    try {
+      backupId = Long.parseLong(args[0]);
+    } catch (final NumberFormatException nfe) {
+      throw new IllegalArgumentException(
+          String.format("Expected as argument the backup ID as long, but got %s", args[0]));
+    }
+
+    try {
+      // Operate
+      final var operateTakeBackupRequestDto =
+          new io.camunda.operate.webapp.management.dto.TakeBackupRequestDto();
+      operateTakeBackupRequestDto.setBackupId(backupId);
+      final var operateBackupResponse =
+          operateBackupManager.takeBackup(operateTakeBackupRequestDto);
+
+      LOG.info(
+          "Triggered ES snapshots for Operate indices: {}",
+          operateBackupResponse.getScheduledSnapshots());
+
+      // Tasklist
+      final var tasklistTakeBackupRequestDto = new TakeBackupRequestDto();
+      tasklistTakeBackupRequestDto.setBackupId(backupId);
+      final var tasklistBackupResponse =
+          tasklistBackupManager.takeBackup(tasklistTakeBackupRequestDto);
+
+      LOG.info(
+          "Triggered ES snapshots for Tasklist indices: {}",
+          tasklistBackupResponse.getScheduledSnapshots());
+    } catch (final Exception e) {
+      LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
+      throw e;
+    }
+
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        throw new IllegalStateException(
+            "Backup with id:[%d] failed.%s".formatted(backupId, failureReason));
+      }
+    }
+
+    LOG.info("Backup with id:[{}] is completed!", backupId);
+  }
+
+  private boolean isCompletedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return operateBackupState == io.camunda.operate.webapp.management.dto.BackupStateDto.COMPLETED
+        && tasklistBackupState == BackupStateDto.COMPLETED;
+  }
+
+  private boolean isFailedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return EnumSet.of(
+                io.camunda.operate.webapp.management.dto.BackupStateDto.FAILED,
+                io.camunda.operate.webapp.management.dto.BackupStateDto.INCOMPATIBLE)
+            .contains(operateBackupState)
+        || EnumSet.of(BackupStateDto.FAILED, BackupStateDto.INCOMPATIBLE)
+            .contains(tasklistBackupState);
+  }
+
+  @SpringBootConfiguration
+  @EnableConfigurationProperties(BrokerBasedProperties.class)
+  @ConfigurationPropertiesScan
+  @ComponentScan(
+      // CAUTION: basePackageClasses makes Spring scan the entire package of the specified
+      // classes, including sub-packages.
+      basePackageClasses = {
+        io.camunda.tasklist.schema.indices.IndexDescriptor.class,
+        io.camunda.tasklist.schema.templates.TemplateDescriptor.class,
+        io.camunda.operate.schema.indices.IndexDescriptor.class,
+        io.camunda.operate.schema.templates.TemplateDescriptor.class,
+        // Backup services from Operate / Tasklist - to be used to create backups
+        io.camunda.operate.webapp.backup.BackupService.class,
+        io.camunda.operate.webapp.elasticsearch.backup.ElasticsearchBackupRepository.class,
+        io.camunda.tasklist.webapp.es.backup.BackupManager.class,
+        // Containing the properties/configurations for Operate/Tasklist
+        io.camunda.operate.property.OperateProperties.class,
+        io.camunda.tasklist.property.TasklistProperties.class,
+        io.camunda.operate.conditions.DatabaseInfo.class,
+        // To set up the right clients, that have to be used by other components
+        io.camunda.operate.connect.ElasticsearchConnector
+            .class, // we need this to find the right clients for Operate
+        io.camunda.tasklist.connect.ElasticsearchConnector.class,
+        // Object mapper used by other components
+        io.camunda.application.commons.sources.DefaultObjectMapperConfiguration.class,
+      },
+      nameGenerator = FullyQualifiedAnnotationBeanNameGenerator.class)
+  public static class BackupManagerConfiguration {
+    @Bean
+    public ExitCodeExceptionMapper exitCodeExceptionMapper() {
+      return ex -> 1;
+    }
+  }
+}
diff --git a/dist/src/main/java/io/camunda/application/StandaloneSchemaManager.java b/dist/src/main/java/io/camunda/application/StandaloneSchemaManager.java
index dc6c97e8edc7..441f26483502 100644
--- a/dist/src/main/java/io/camunda/application/StandaloneSchemaManager.java
+++ b/dist/src/main/java/io/camunda/application/StandaloneSchemaManager.java
@@ -149,6 +149,7 @@ public void run(final String... args) throws Exception {
         io.camunda.tasklist.es.RetryElasticsearchClient.class,
         io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.class,
         DefaultObjectMapperConfiguration.class,
+        io.camunda.tasklist.connect.ElasticsearchConnector.class,
       },
       nameGenerator = FullyQualifiedAnnotationBeanNameGenerator.class)
   public static class SchemaManagerConfiguration {
diff --git a/operate/schema/src/main/java/io/camunda/operate/store/elasticsearch/ElasticsearchDecisionStore.java b/operate/schema/src/main/java/io/camunda/operate/store/elasticsearch/ElasticsearchDecisionStore.java
index 83b9b1fbf714..f4d433fbaadb 100644
--- a/operate/schema/src/main/java/io/camunda/operate/store/elasticsearch/ElasticsearchDecisionStore.java
+++ b/operate/schema/src/main/java/io/camunda/operate/store/elasticsearch/ElasticsearchDecisionStore.java
@@ -13,7 +13,6 @@
 import io.camunda.operate.schema.indices.DecisionIndex;
 import io.camunda.operate.store.BatchRequest;
 import io.camunda.operate.store.DecisionStore;
-import io.camunda.operate.tenant.TenantAwareElasticsearchClient;
 import java.io.IOException;
 import java.util.Optional;
 import org.elasticsearch.action.search.SearchRequest;
@@ -46,10 +45,8 @@ public class ElasticsearchDecisionStore implements DecisionStore {
 
   @Autowired private RestHighLevelClient esClient;
 
-  @Autowired private TenantAwareElasticsearchClient tenantAwareClient;
-
   @Override
-  public Optional<Long> getDistinctCountFor(String fieldName) {
+  public Optional<Long> getDistinctCountFor(final String fieldName) {
     final String indexAlias = decisionIndex.getAlias();
     LOGGER.debug("Called distinct count for field {} in index alias {}.", fieldName, indexAlias);
     final SearchRequest searchRequest =
@@ -67,7 +64,7 @@ public Optional<Long> getDistinctCountFor(String fieldName) {
       final Cardinality distinctFieldCounts =
           searchResponse.getAggregations().get(DISTINCT_FIELD_COUNTS);
       return Optional.of(distinctFieldCounts.getValue());
-    } catch (Exception e) {
+    } catch (final Exception e) {
       LOGGER.error(
           String.format(
               "Error in distinct count for field %s in index alias %s.", fieldName, indexAlias),
@@ -82,7 +79,8 @@ public BatchRequest newBatchRequest() {
   }
 
   @Override
-  public long deleteDocuments(String indexName, String idField, String id) throws IOException {
+  public long deleteDocuments(final String indexName, final String idField, final String id)
+      throws IOException {
     final DeleteByQueryRequest query =
         new DeleteByQueryRequest(indexName).setQuery(QueryBuilders.termsQuery(idField, id));
     final BulkByScrollResponse response = esClient.deleteByQuery(query, RequestOptions.DEFAULT);
diff --git a/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDetailDto.java b/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDetailDto.java
index 6f6d7f6c3bb5..f25f09f624c5 100644
--- a/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDetailDto.java
+++ b/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDetailDto.java
@@ -24,7 +24,7 @@ public String getSnapshotName() {
     return snapshotName;
   }
 
-  public GetBackupStateResponseDetailDto setSnapshotName(String snapshotName) {
+  public GetBackupStateResponseDetailDto setSnapshotName(final String snapshotName) {
     this.snapshotName = snapshotName;
     return this;
   }
@@ -33,7 +33,7 @@ public String getState() {
     return state;
   }
 
-  public GetBackupStateResponseDetailDto setState(String state) {
+  public GetBackupStateResponseDetailDto setState(final String state) {
     this.state = state;
     return this;
   }
@@ -42,7 +42,7 @@ public OffsetDateTime getStartTime() {
     return startTime;
   }
 
-  public GetBackupStateResponseDetailDto setStartTime(OffsetDateTime startTime) {
+  public GetBackupStateResponseDetailDto setStartTime(final OffsetDateTime startTime) {
     this.startTime = startTime;
     return this;
   }
@@ -51,7 +51,7 @@ public String[] getFailures() {
     return failures;
   }
 
-  public GetBackupStateResponseDetailDto setFailures(String[] failures) {
+  public GetBackupStateResponseDetailDto setFailures(final String[] failures) {
     this.failures = failures;
     return this;
   }
@@ -64,7 +64,7 @@ public int hashCode() {
   }
 
   @Override
-  public boolean equals(Object o) {
+  public boolean equals(final Object o) {
     if (this == o) {
       return true;
     }
@@ -77,4 +77,20 @@ public boolean equals(Object o) {
         && Objects.equals(startTime, that.startTime)
         && Arrays.equals(failures, that.failures);
   }
+
+  @Override
+  public String toString() {
+    return "GetBackupStateResponseDetailDto{"
+        + "snapshotName='"
+        + snapshotName
+        + '\''
+        + ", state='"
+        + state
+        + '\''
+        + ", startTime="
+        + startTime
+        + ", failures="
+        + Arrays.toString(failures)
+        + '}';
+  }
 }
diff --git a/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDto.java b/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDto.java
index d2a6765e32a7..1aee4671d2a3 100644
--- a/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDto.java
+++ b/operate/webapp/src/main/java/io/camunda/operate/webapp/management/dto/GetBackupStateResponseDto.java
@@ -19,7 +19,7 @@ public class GetBackupStateResponseDto {
 
   public GetBackupStateResponseDto() {}
 
-  public GetBackupStateResponseDto(Long backupId) {
+  public GetBackupStateResponseDto(final Long backupId) {
     this.backupId = backupId;
   }
 
@@ -27,7 +27,7 @@ public Long getBackupId() {
     return backupId;
   }
 
-  public GetBackupStateResponseDto setBackupId(Long backupId) {
+  public GetBackupStateResponseDto setBackupId(final Long backupId) {
     this.backupId = backupId;
     return this;
   }
@@ -36,7 +36,7 @@ public BackupStateDto getState() {
     return state;
   }
 
-  public GetBackupStateResponseDto setState(BackupStateDto state) {
+  public GetBackupStateResponseDto setState(final BackupStateDto state) {
     this.state = state;
     return this;
   }
@@ -45,7 +45,7 @@ public String getFailureReason() {
     return failureReason;
   }
 
-  public GetBackupStateResponseDto setFailureReason(String failureReason) {
+  public GetBackupStateResponseDto setFailureReason(final String failureReason) {
     this.failureReason = failureReason;
     return this;
   }
@@ -54,7 +54,7 @@ public List<GetBackupStateResponseDetailDto> getDetails() {
     return details;
   }
 
-  public GetBackupStateResponseDto setDetails(List<GetBackupStateResponseDetailDto> details) {
+  public GetBackupStateResponseDto setDetails(final List<GetBackupStateResponseDetailDto> details) {
     this.details = details;
     return this;
   }
@@ -65,7 +65,7 @@ public int hashCode() {
   }
 
   @Override
-  public boolean equals(Object o) {
+  public boolean equals(final Object o) {
     if (this == o) {
       return true;
     }
@@ -78,4 +78,19 @@ public boolean equals(Object o) {
         && Objects.equals(failureReason, that.failureReason)
         && Objects.equals(details, that.details);
   }
+
+  @Override
+  public String toString() {
+    return "GetBackupStateResponseDto{"
+        + "backupId="
+        + backupId
+        + ", state="
+        + state
+        + ", failureReason='"
+        + failureReason
+        + '\''
+        + ", details="
+        + details
+        + '}';
+  }
 }
diff --git a/qa/integration-tests/pom.xml b/qa/integration-tests/pom.xml
index cc233662afd6..574dad721a60 100644
--- a/qa/integration-tests/pom.xml
+++ b/qa/integration-tests/pom.xml
@@ -194,6 +194,11 @@
       <artifactId>operate-common</artifactId>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>io.camunda</groupId>
+      <artifactId>tasklist-webapp</artifactId>
+      <scope>test</scope>
+    </dependency>
   </dependencies>
 
   <build>
diff --git a/qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java b/qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
new file mode 100644
index 000000000000..d1157b86c160
--- /dev/null
+++ b/qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
@@ -0,0 +1,450 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
+          // Enable security features
+          .withEnv("xpack.security.enabled", "true")
+          // Ensure a fast and reliable startup
+          .withStartupTimeout(Duration.ofMinutes(5))
+          .withStartupAttempts(3)
+          .withEnv("xpack.security.enabled", "true")
+          .withEnv("xpack.watcher.enabled", "false")
+          .withEnv("xpack.ml.enabled", "false")
+          // Configure with allowed repository storage path
+          .withEnv("path.repo", "~/")
+          // to be able to delete indices with wildcards
+          .withEnv("action.destructive_requires_name", "false")
+          .withCopyToContainer(
+              Transferable.of(APP_ROLE_DEFINITION), "/usr/share/elasticsearch/config/roles.yml")
+          .withClasspathResourceMapping(
+              "elasticsearch-fast-startup.options",
+              "/usr/share/elasticsearch/config/jvm.options.d/elasticsearch-fast-startup.options",
+              BindMode.READ_ONLY);
+
+  // Configure the Camunda single application with restricted access to the Elasticsearch
+  @TestZeebe(autoStart = false)
+  final TestStandaloneCamunda camunda =
+      new TestStandaloneCamunda(es, false)
+          .withAdditionalProfile(Profile.DEFAULT_AUTH_PROFILE)
+          .withProperty("camunda.operate.migration.migrationEnabled", false)
+          .withProperty("camunda.tasklist.migration.migrationEnabled", false)
+          .withProperty("camunda.database.username", APP_USER)
+          .withProperty("camunda.database.password", APP_PASSWORD)
+          .withBrokerConfig(
+              cfg -> {
+                cfg.getExporters()
+                    .computeIfAbsent("elasticsearch", __ -> new ExporterCfg())
+                    .setArgs(
+                        Map.of(
+                            "index",
+                            Map.of("createTemplate", false),
+                            "retention",
+                            Map.of("enabled", false, "managePolicy", false),
+                            "authentication",
+                            Map.of("username", APP_USER, "password", APP_PASSWORD)));
+              })
+          .withOperateConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
+              })
+          .withTasklistConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
+                cfg.getArchiver().setIlmManagePolicy(false);
+              });
+
+  // Elasticsearch client with admin access to apply some manual operations
+  private ElasticsearchClient adminElasticsearchClient;
+
+  @BeforeEach
+  void setup() throws IOException, InterruptedException {
+    // setup ES users
+    es.execInContainer("elasticsearch-users", "useradd", APP_USER, "-p", APP_PASSWORD);
+    es.execInContainer("elasticsearch-users", "useradd", ADMIN_USER, "-p", ADMIN_PASSWORD);
+    es.execInContainer("elasticsearch-users", "roles", ADMIN_USER, "-a", ADMIN_ROLE);
+    es.execInContainer("elasticsearch-users", "roles", APP_USER, "-a", APP_ROLE);
+    // Connect to ES
+    schemaManager.withProperty(
+        "zeebe.broker.exporters.elasticsearch.args.url", "http://" + es.getHttpHostAddress());
+    schemaManager.withProperty(
+        "camunda.operate.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    schemaManager.withProperty(
+        "camunda.tasklist.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    backupManager.withProperty(
+        "camunda.operate.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    backupManager.withProperty(
+        "camunda.tasklist.elasticsearch.url", "http://" + es.getHttpHostAddress());
+
+    adminElasticsearchClient = createAdminElasticsearchClient("http://" + es.getHttpHostAddress());
+  }
+
+  @Test
+  void canBackupRestore() throws Exception {
+    // GIVEN
+    // create the schema
+    schemaManager.start();
+    // Create a snapshot repository to store backups
+    createSnapshotRepository();
+    // Start the Camunda application
+    camunda.start();
+    // Generate test data
+    final long processInstanceKey = generateData();
+    // Assert that the data is created in Elasticsearch
+    final long userTaskKey =
+        assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance());
+
+    // WHEN
+    // Start the backup process with a specific backup ID
+    backupManager.withBackupId(BACKUP_ID).start();
+    // manually backup zeebe indices
+    backupZeebeIndices();
+
+    final List<String> snapshots = new ArrayList<>();
+    // Wait for snapshots to be completed
+    snapshots.addAll(waitForSnapshotsToBeCompleted(OPERATE_SNAPSHOT_NAME_PREFIX, 6));
+    snapshots.addAll(waitForSnapshotsToBeCompleted(TASKLIST_SNAPSHOT_NAME_PREFIX, 6));
+    snapshots.addAll(waitForSnapshotsToBeCompleted(ZEEBE_SNAPSHOT_NAME, 1));
+    // Update the current state by completing the user task and the process instance
+    completeUserTask(userTaskKey);
+    // Assert that the state is updated: process instance is completed
+    assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance().negate());
+
+    // Stop the Camunda application
+    camunda.stop();
+    // Delete indices to simulate a fresh start
+    deleteIndices();
+    // Restore the backup to recover deleted data
+    restoreBackup(snapshots);
+    // Restart Camunda after restoration
+    camunda.start();
+
+    // THEN
+    // Validate that the data is restored successfully at the state before the backup: process
+    // instance is running
+    assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance());
+  }
+
+  /**
+   * As documented in
+   * https://docs.camunda.io/docs/self-managed/operational-guides/backup-restore/backup-and-restore/#backup-process
+   */
+  private void backupZeebeIndices() throws IOException {
+    adminElasticsearchClient
+        .snapshot()
+        .create(
+            r ->
+                r.repository(REPOSITORY_NAME)
+                    .snapshot(ZEEBE_SNAPSHOT_NAME)
+                    .indices("zeebe-record*")
+                    .featureStates("none"));
+  }
+
+  private long generateData() {
+    // creating a user for webapps authentication
+    try (final var operateClient = camunda.newOperateClient()) {
+      operateClient.createUser(TEST_USER_NAME, TEST_USER_PASSWORD);
+    }
+    // creating a process instance with user task
+    final long processInstanceKey;
+    try (final var zeebeClient = camunda.newClientBuilder().build()) {
+      // Deploy process with user task
+      zeebeClient
+          .newDeployResourceCommand()
+          .addProcessModel(
+              Bpmn.createExecutableProcess("process-with-user-task")
+                  .startEvent()
+                  .userTask("user-task")
+                  .zeebeUserTask()
+                  .zeebeAssignee(TEST_USER_NAME)
+                  .endEvent()
+                  .done(),
+              "process-with-user-task.bpmn")
+          .send()
+          .join();
+      processInstanceKey =
+          zeebeClient
+              .newCreateInstanceCommand()
+              .bpmnProcessId("process-with-user-task")
+              .latestVersion()
+              .send()
+              .join()
+              .getProcessInstanceKey();
+    }
+    return processInstanceKey;
+  }
+
+  private List<String> waitForSnapshotsToBeCompleted(
+      final String snapshotNamePrefix, final int count) {
+    final var snapshots = new ArrayList<String>();
+    Awaitility.await("should find all snapshots completed")
+        .atMost(ofSeconds(30))
+        .pollDelay(ofSeconds(2))
+        .untilAsserted(
+            () -> {
+              final var response =
+                  adminElasticsearchClient
+                      .snapshot()
+                      .get(r -> r.repository(REPOSITORY_NAME).snapshot(snapshotNamePrefix + "*"))
+                      .snapshots();
+              assertThat(response).hasSize(count);
+              assertThat(response.stream().map(snapshot -> snapshot.state()))
+                  .allMatch(state -> "SUCCESS".equals(state));
+              snapshots.addAll(response.stream().map(snapshot -> snapshot.snapshot()).toList());
+            });
+    return snapshots;
+  }
+
+  /**
+   * Asserts that the data is present in Operate and Tasklist.
+   *
+   * <ul>
+   *   <li>Asserts that the process instance is present in Operate and verifies the check.
+   *   <li>Asserts that one user task is present in Tasklist and returns its key.
+   * </ul>
+   */
+  private long assertThatDataIsPresent(
+      final long processInstanceKey, final Predicate<ProcessInstance> processInstanceCheck) {
+    try (final var operateClient = camunda.newOperateClient()) {
+      Awaitility.await("should find a process instance")
+          .atMost(ofSeconds(30))
+          .ignoreExceptions()
+          .untilAsserted(
+              () -> {
+                final var result =
+                    operateClient
+                        .withAuthentication(TEST_USER_NAME, TEST_USER_PASSWORD)
+                        .getProcessInstanceWith(processInstanceKey);
+                EitherAssert.assertThat(result)
+                    .right()
+                    .isNotNull()
+                    .extracting(ProcessInstanceResult::total)
+                    .isEqualTo(1L);
+                final var processInstance = result.get().processInstances().getFirst();
+                assertThat(processInstance).matches(processInstanceCheck);
+              });
+    }
+    final var userTaskKey = new AtomicReference<Long>();
+    try (final var tasklistClient = camunda.newTasklistClient()) {
+      Awaitility.await("should find a user task")
+          .atMost(ofSeconds(30))
+          .ignoreExceptions()
+          .untilAsserted(
+              () -> {
+                final List<SearchQueryHit<TaskEntity>> hits =
+                    tasklistClient
+                        .withAuthentication(TEST_USER_NAME, TEST_USER_PASSWORD)
+                        .searchUserTasks(SearchQueryBuilders.query().build())
+                        .hits();
+                assertThat(hits).hasSize(1);
+                userTaskKey.set(hits.get(0).source().getKey());
+              });
+    }
+    return userTaskKey.get();
+  }
+
+  private void deleteIndices() throws IOException {
+    adminElasticsearchClient
+        .indices()
+        .delete(r -> r.index("operate*", "tasklist*", "zeebe*").ignoreUnavailable(true));
+    Awaitility.await("should delete indices")
+        .atMost(ofSeconds(30))
+        .pollDelay(ofSeconds(2))
+        .untilAsserted(
+            () ->
+                assertThat(
+                        adminElasticsearchClient
+                            .indices()
+                            .exists(
+                                r ->
+                                    r.index("operate*", "tasklist*", "zeebe*")
+                                        .allowNoIndices(false))
+                            .value())
+                    .isFalse());
+  }
+
+  /**
+   * As documented in
+   * https://docs.camunda.io/docs/self-managed/operational-guides/backup-restore/backup-and-restore/#restore
+   */
+  private void restoreBackup(final List<String> snapshots) {
+    snapshots.stream()
+        .forEach(
+            snapshot -> {
+              try {
+                adminElasticsearchClient
+                    .snapshot()
+                    .restore(
+                        r ->
+                            r.repository(REPOSITORY_NAME)
+                                .snapshot(snapshot)
+                                .waitForCompletion(true));
+              } catch (final IOException e) {
+                fail("Exception occurred while restoring the backup: " + e.getMessage(), e);
+              }
+            });
+  }
+
+  private void createSnapshotRepository() throws IOException {
+    adminElasticsearchClient
+        .snapshot()
+        .createRepository(
+            q ->
+                q.name(REPOSITORY_NAME)
+                    .repository(r -> r.fs(fs -> fs.settings(s -> s.location(REPOSITORY_NAME)))));
+  }
+
+  private void completeUserTask(final long userTaskKey) {
+    try (final var tasklistClient = camunda.newTasklistClient()) {
+      assertThat(
+              tasklistClient
+                  .withAuthentication(TEST_USER_NAME, TEST_USER_PASSWORD)
+                  .completeUserTask(userTaskKey))
+          .returns(200, HttpResponse::statusCode);
+    }
+  }
+
+  private ElasticsearchClient createAdminElasticsearchClient(final String elasticSearchUrl) {
+    final var connectConfiguration = new ConnectConfiguration();
+    connectConfiguration.setUrl(elasticSearchUrl);
+    connectConfiguration.setUsername(ADMIN_USER);
+    connectConfiguration.setPassword(ADMIN_PASSWORD);
+    return new ElasticsearchConnector(connectConfiguration).createClient();
+  }
+
+  private static Predicate<ProcessInstance> isRunningProcessInstance() {
+    return processInstance -> processInstance.getEndDate() == null;
+  }
+}
diff --git a/qa/util/src/main/java/io/camunda/qa/util/cluster/TestStandaloneBackupManager.java b/qa/util/src/main/java/io/camunda/qa/util/cluster/TestStandaloneBackupManager.java
new file mode 100644
index 000000000000..9ead019a91eb
--- /dev/null
+++ b/qa/util/src/main/java/io/camunda/qa/util/cluster/TestStandaloneBackupManager.java
@@ -0,0 +1,66 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.qa.util.cluster;
+
+import io.atomix.cluster.MemberId;
+import io.camunda.application.StandaloneBackupManager;
+import io.camunda.application.StandaloneBackupManager.BackupManagerConfiguration;
+import io.camunda.zeebe.qa.util.actuator.HealthActuator;
+import io.camunda.zeebe.qa.util.actuator.HealthActuator.NoopHealthActuator;
+import io.camunda.zeebe.qa.util.cluster.TestSpringApplication;
+import org.springframework.boot.WebApplicationType;
+import org.springframework.boot.builder.SpringApplicationBuilder;
+
+public class TestStandaloneBackupManager
+    extends TestSpringApplication<TestStandaloneBackupManager> {
+
+  private Long backupId;
+
+  public TestStandaloneBackupManager() {
+    super(
+        BackupManagerConfiguration.class,
+        StandaloneBackupManager.class,
+        io.camunda.tasklist.JacksonConfig.class,
+        io.camunda.operate.JacksonConfig.class);
+  }
+
+  @Override
+  public TestStandaloneBackupManager self() {
+    return this;
+  }
+
+  @Override
+  public MemberId nodeId() {
+    return MemberId.from("backup");
+  }
+
+  @Override
+  public HealthActuator healthActuator() {
+    return new NoopHealthActuator();
+  }
+
+  @Override
+  public boolean isGateway() {
+    return false;
+  }
+
+  @Override
+  protected String[] commandLineArgs() {
+    return backupId == null ? super.commandLineArgs() : new String[] {String.valueOf(backupId)};
+  }
+
+  @Override
+  protected SpringApplicationBuilder createSpringBuilder() {
+    return super.createSpringBuilder().web(WebApplicationType.NONE);
+  }
+
+  public TestStandaloneBackupManager withBackupId(final Long backupId) {
+    this.backupId = backupId;
+    return this;
+  }
+}
diff --git a/tasklist/common/src/main/java/io/camunda/tasklist/CommonUtils.java b/tasklist/common/src/main/java/io/camunda/tasklist/CommonUtils.java
index 9a26bfaa8f26..16025efb9cd3 100644
--- a/tasklist/common/src/main/java/io/camunda/tasklist/CommonUtils.java
+++ b/tasklist/common/src/main/java/io/camunda/tasklist/CommonUtils.java
@@ -18,7 +18,7 @@
 import com.fasterxml.jackson.databind.SerializationFeature;
 import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
-import io.camunda.tasklist.es.ElasticsearchConnector;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import jakarta.json.Json;
 import jakarta.json.JsonObject;
 import jakarta.json.JsonObjectBuilder;
@@ -56,11 +56,11 @@ public static ObjectMapper getObjectMapper() {
         .build();
   }
 
-  public static JsonObject getJsonObjectFromEntity(Object o) {
+  public static JsonObject getJsonObjectFromEntity(final Object o) {
     return Json.createObjectBuilder(OBJECT_MAPPER.convertValue(o, HashMap.class)).build();
   }
 
-  public static JsonObjectBuilder getJsonObjectBuilderForEntity(Object o) {
+  public static JsonObjectBuilder getJsonObjectBuilderForEntity(final Object o) {
     return Json.createObjectBuilder(OBJECT_MAPPER.convertValue(o, HashMap.class));
   }
 }
diff --git a/tasklist/common/src/main/java/io/camunda/tasklist/JacksonConfig.java b/tasklist/common/src/main/java/io/camunda/tasklist/JacksonConfig.java
index 12fc8cb4dda0..6a4cc3d27d66 100644
--- a/tasklist/common/src/main/java/io/camunda/tasklist/JacksonConfig.java
+++ b/tasklist/common/src/main/java/io/camunda/tasklist/JacksonConfig.java
@@ -13,7 +13,7 @@
 import com.fasterxml.jackson.databind.SerializationFeature;
 import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
-import io.camunda.tasklist.es.ElasticsearchConnector;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import io.camunda.tasklist.property.TasklistProperties;
 import java.time.Instant;
 import java.time.OffsetDateTime;
diff --git a/tasklist/common/src/main/java/io/camunda/tasklist/es/ElasticsearchConnector.java b/tasklist/common/src/main/java/io/camunda/tasklist/connect/ElasticsearchConnector.java
similarity index 99%
rename from tasklist/common/src/main/java/io/camunda/tasklist/es/ElasticsearchConnector.java
rename to tasklist/common/src/main/java/io/camunda/tasklist/connect/ElasticsearchConnector.java
index 3f2de06cd233..96b7bde9f13f 100644
--- a/tasklist/common/src/main/java/io/camunda/tasklist/es/ElasticsearchConnector.java
+++ b/tasklist/common/src/main/java/io/camunda/tasklist/connect/ElasticsearchConnector.java
@@ -5,7 +5,7 @@
  * Licensed under the Camunda License 1.0. You may not use this file
  * except in compliance with the Camunda License 1.0.
  */
-package io.camunda.tasklist.es;
+package io.camunda.tasklist.connect;
 
 import co.elastic.clients.elasticsearch.ElasticsearchClient;
 import co.elastic.clients.elasticsearch.cluster.HealthResponse;
diff --git a/tasklist/common/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorTest.java b/tasklist/common/src/test/java/io/camunda/tasklist/connect/ElasticsearchConnectorTest.java
similarity index 99%
rename from tasklist/common/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorTest.java
rename to tasklist/common/src/test/java/io/camunda/tasklist/connect/ElasticsearchConnectorTest.java
index c07626c9c131..f770b8958c07 100644
--- a/tasklist/common/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorTest.java
+++ b/tasklist/common/src/test/java/io/camunda/tasklist/connect/ElasticsearchConnectorTest.java
@@ -5,7 +5,7 @@
  * Licensed under the Camunda License 1.0. You may not use this file
  * except in compliance with the Camunda License 1.0.
  */
-package io.camunda.tasklist.es;
+package io.camunda.tasklist.connect;
 
 import static org.assertj.core.api.Assertions.assertThat;
 
diff --git a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorIT.java b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorIT.java
index df445ba14b69..c0c1825208dd 100644
--- a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorIT.java
+++ b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorIT.java
@@ -12,6 +12,7 @@
 import com.github.tomakehurst.wiremock.client.WireMock;
 import com.github.tomakehurst.wiremock.core.WireMockConfiguration;
 import io.camunda.search.connect.plugin.PluginConfiguration;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import io.camunda.tasklist.property.TasklistProperties;
 import io.camunda.tasklist.util.TestPlugin;
 import io.camunda.zeebe.util.FileUtil;
diff --git a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorSSLAuthIT.java b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorSSLAuthIT.java
index 8093981230d4..2140b17f8057 100644
--- a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorSSLAuthIT.java
+++ b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/es/ElasticsearchConnectorSSLAuthIT.java
@@ -10,6 +10,7 @@
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.junit.jupiter.api.Assumptions.assumeTrue;
 
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import io.camunda.tasklist.property.TasklistProperties;
 import io.camunda.tasklist.qa.util.TestUtil;
 import io.camunda.tasklist.util.TasklistIntegrationTest;
@@ -83,7 +84,7 @@ static class ElasticsearchStarter
       implements ApplicationContextInitializer<ConfigurableApplicationContext> {
 
     @Override
-    public void initialize(ConfigurableApplicationContext applicationContext) {
+    public void initialize(final ConfigurableApplicationContext applicationContext) {
       elasticsearch.start();
 
       final String elsUrl =
diff --git a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckAuthenticationIT.java b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckAuthenticationIT.java
index 7e4efc065077..9f36af757a72 100644
--- a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckAuthenticationIT.java
+++ b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckAuthenticationIT.java
@@ -14,7 +14,7 @@
 import static org.mockito.BDDMockito.given;
 
 import io.camunda.tasklist.JacksonConfig;
-import io.camunda.tasklist.es.ElasticsearchConnector;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import io.camunda.tasklist.es.ElasticsearchInternalTask;
 import io.camunda.tasklist.es.RetryElasticsearchClient;
 import io.camunda.tasklist.property.TasklistProperties;
diff --git a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckIT.java b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckIT.java
index c28efb5b4c1b..ababd4048008 100644
--- a/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckIT.java
+++ b/tasklist/qa/integration-tests/src/test/java/io/camunda/tasklist/management/HealthCheckIT.java
@@ -15,7 +15,7 @@
 import static org.mockito.Mockito.verifyNoInteractions;
 
 import io.camunda.tasklist.JacksonConfig;
-import io.camunda.tasklist.es.ElasticsearchConnector;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import io.camunda.tasklist.es.ElasticsearchInternalTask;
 import io.camunda.tasklist.es.RetryElasticsearchClient;
 import io.camunda.tasklist.property.TasklistProperties;
diff --git a/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDetailDto.java b/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDetailDto.java
index fe90412a8d35..06ec0c75f61c 100644
--- a/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDetailDto.java
+++ b/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDetailDto.java
@@ -24,7 +24,7 @@ public String getSnapshotName() {
     return snapshotName;
   }
 
-  public GetBackupStateResponseDetailDto setSnapshotName(String snapshotName) {
+  public GetBackupStateResponseDetailDto setSnapshotName(final String snapshotName) {
     this.snapshotName = snapshotName;
     return this;
   }
@@ -33,7 +33,7 @@ public String getState() {
     return state;
   }
 
-  public GetBackupStateResponseDetailDto setState(String state) {
+  public GetBackupStateResponseDetailDto setState(final String state) {
     this.state = state;
     return this;
   }
@@ -42,7 +42,7 @@ public OffsetDateTime getStartTime() {
     return startTime;
   }
 
-  public GetBackupStateResponseDetailDto setStartTime(OffsetDateTime startTime) {
+  public GetBackupStateResponseDetailDto setStartTime(final OffsetDateTime startTime) {
     this.startTime = startTime;
     return this;
   }
@@ -51,13 +51,20 @@ public String[] getFailures() {
     return failures;
   }
 
-  public GetBackupStateResponseDetailDto setFailures(String[] failures) {
+  public GetBackupStateResponseDetailDto setFailures(final String[] failures) {
     this.failures = failures;
     return this;
   }
 
   @Override
-  public boolean equals(Object o) {
+  public int hashCode() {
+    int result = Objects.hash(snapshotName, state, startTime);
+    result = 31 * result + Arrays.hashCode(failures);
+    return result;
+  }
+
+  @Override
+  public boolean equals(final Object o) {
     if (this == o) {
       return true;
     }
@@ -72,9 +79,18 @@ public boolean equals(Object o) {
   }
 
   @Override
-  public int hashCode() {
-    int result = Objects.hash(snapshotName, state, startTime);
-    result = 31 * result + Arrays.hashCode(failures);
-    return result;
+  public String toString() {
+    return "GetBackupStateResponseDetailDto{"
+        + "snapshotName='"
+        + snapshotName
+        + '\''
+        + ", state='"
+        + state
+        + '\''
+        + ", startTime="
+        + startTime
+        + ", failures="
+        + Arrays.toString(failures)
+        + '}';
   }
 }
diff --git a/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDto.java b/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDto.java
index 9674a853f690..b65f14baf7b3 100644
--- a/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDto.java
+++ b/tasklist/webapp/src/main/java/io/camunda/tasklist/webapp/management/dto/GetBackupStateResponseDto.java
@@ -19,7 +19,7 @@ public class GetBackupStateResponseDto {
 
   public GetBackupStateResponseDto() {}
 
-  public GetBackupStateResponseDto(Long backupId) {
+  public GetBackupStateResponseDto(final Long backupId) {
     this.backupId = backupId;
   }
 
@@ -27,7 +27,7 @@ public Long getBackupId() {
     return backupId;
   }
 
-  public GetBackupStateResponseDto setBackupId(Long backupId) {
+  public GetBackupStateResponseDto setBackupId(final Long backupId) {
     this.backupId = backupId;
     return this;
   }
@@ -36,7 +36,7 @@ public BackupStateDto getState() {
     return state;
   }
 
-  public GetBackupStateResponseDto setState(BackupStateDto state) {
+  public GetBackupStateResponseDto setState(final BackupStateDto state) {
     this.state = state;
     return this;
   }
@@ -45,7 +45,7 @@ public String getFailureReason() {
     return failureReason;
   }
 
-  public GetBackupStateResponseDto setFailureReason(String failureReason) {
+  public GetBackupStateResponseDto setFailureReason(final String failureReason) {
     this.failureReason = failureReason;
     return this;
   }
@@ -54,13 +54,18 @@ public List<GetBackupStateResponseDetailDto> getDetails() {
     return details;
   }
 
-  public GetBackupStateResponseDto setDetails(List<GetBackupStateResponseDetailDto> details) {
+  public GetBackupStateResponseDto setDetails(final List<GetBackupStateResponseDetailDto> details) {
     this.details = details;
     return this;
   }
 
   @Override
-  public boolean equals(Object o) {
+  public int hashCode() {
+    return Objects.hash(backupId, state, failureReason, details);
+  }
+
+  @Override
+  public boolean equals(final Object o) {
     if (this == o) {
       return true;
     }
@@ -75,7 +80,17 @@ public boolean equals(Object o) {
   }
 
   @Override
-  public int hashCode() {
-    return Objects.hash(backupId, state, failureReason, details);
+  public String toString() {
+    return "GetBackupStateResponseDto{"
+        + "backupId="
+        + backupId
+        + ", state="
+        + state
+        + ", failureReason='"
+        + failureReason
+        + '\''
+        + ", details="
+        + details
+        + '}';
   }
 }
diff --git a/tasklist/webapp/src/test/java/io/camunda/tasklist/webapp/CommonUtils.java b/tasklist/webapp/src/test/java/io/camunda/tasklist/webapp/CommonUtils.java
index 9ed194f6742c..75b6d387ced1 100644
--- a/tasklist/webapp/src/test/java/io/camunda/tasklist/webapp/CommonUtils.java
+++ b/tasklist/webapp/src/test/java/io/camunda/tasklist/webapp/CommonUtils.java
@@ -13,7 +13,7 @@
 import com.fasterxml.jackson.databind.SerializationFeature;
 import com.fasterxml.jackson.datatype.jdk8.Jdk8Module;
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
-import io.camunda.tasklist.es.ElasticsearchConnector;
+import io.camunda.tasklist.connect.ElasticsearchConnector;
 import java.time.Instant;
 import java.time.OffsetDateTime;
 import java.time.format.DateTimeFormatter;


Reviews:
user: login: ChrisKujawa
body: Thanks @houssain-barouni looks good, I had only some smaller notes/comments, and some questions. See below
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T14:09:17Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:12:14Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:13:48Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:16:07Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:17:33Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:18:18Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T14:27:29Z
user: login: houssain-barouni
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T15:46:54Z
user: login: houssain-barouni
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-10T16:58:57Z

Review comments:
diff_hunk: @@ -0,0 +1,66 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.qa.util.cluster;
+
+import io.atomix.cluster.MemberId;
+import io.camunda.application.StandaloneBackupManager;
+import io.camunda.application.StandaloneBackupManager.BackupManagerConfiguration;
+import io.camunda.zeebe.qa.util.actuator.HealthActuator;
+import io.camunda.zeebe.qa.util.actuator.HealthActuator.NoopHealthActuator;
+import io.camunda.zeebe.qa.util.cluster.TestSpringApplication;
+import org.springframework.boot.WebApplicationType;
+import org.springframework.boot.builder.SpringApplicationBuilder;
+
+public class TestStandaloneBackupManager
+    extends TestSpringApplication<TestStandaloneBackupManager> {
+
+  private Long backupId;
+
+  public TestStandaloneBackupManager() {
+    super(
+        BackupManagerConfiguration.class,
+        StandaloneBackupManager.class,
+        io.camunda.tasklist.JacksonConfig.class,
+        io.camunda.operate.JacksonConfig.class);
+  }
+
+  @Override
+  public TestStandaloneBackupManager self() {
+    return this;
+  }
+
+  @Override
+  public MemberId nodeId() {
+    return MemberId.from("backup");
+  }
+
+  @Override
+  public HealthActuator healthActuator() {
+    return new NoopHealthActuator();
+  }
+
+  @Override
+  public boolean isGateway() {
+    return false;
+  }
+
+  @Override
+  protected String[] commandLineArgs() {
+    return backupId == null ? super.commandLineArgs() : new String[] {String.valueOf(backupId)};
+  }
+
path: qa/util/src/main/java/io/camunda/qa/util/cluster/TestStandaloneBackupManager.java
user: login: ChrisKujawa
body: TIL 👍🏼 
created_at: 2025-03-10T13:49:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: ChrisKujawa
body: ❓ Do we know why we have this? I know it was already set on the schema manager examples and test, but wondering why. Is it to disable Operate failing or waiting for ES to become available? 🤔 
created_at: 2025-03-10T13:51:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: ChrisKujawa
body: 🙃 Just sharing my recent learning or pain with this. 

When migrating to ES java client 8.16.4 this approach fails, because the artifact has no details in their metainf 🤣 🙈 
created_at: 2025-03-10T13:53:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: ChrisKujawa
body: https://github.com/camunda/camunda/pull/29167#discussion_r1982062746
created_at: 2025-03-10T13:53:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
+          // Enable security features
+          .withEnv("xpack.security.enabled", "true")
+          // Ensure a fast and reliable startup
+          .withStartupTimeout(Duration.ofMinutes(5))
+          .withStartupAttempts(3)
+          .withEnv("xpack.security.enabled", "true")
+          .withEnv("xpack.watcher.enabled", "false")
+          .withEnv("xpack.ml.enabled", "false")
+          // Configure with allowed repository storage path
+          .withEnv("path.repo", "~/")
+          // to be able to delete indices with wildcards
+          .withEnv("action.destructive_requires_name", "false")
+          .withCopyToContainer(
+              Transferable.of(APP_ROLE_DEFINITION), "/usr/share/elasticsearch/config/roles.yml")
+          .withClasspathResourceMapping(
+              "elasticsearch-fast-startup.options",
+              "/usr/share/elasticsearch/config/jvm.options.d/elasticsearch-fast-startup.options",
+              BindMode.READ_ONLY);
+
+  // Configure the Camunda single application with restricted access to the Elasticsearch
+  @TestZeebe(autoStart = false)
+  final TestStandaloneCamunda camunda =
+      new TestStandaloneCamunda(es, false)
+          .withAdditionalProfile(Profile.DEFAULT_AUTH_PROFILE)
+          .withProperty("camunda.operate.migration.migrationEnabled", false)
+          .withProperty("camunda.tasklist.migration.migrationEnabled", false)
+          .withProperty("camunda.database.username", APP_USER)
+          .withProperty("camunda.database.password", APP_PASSWORD)
+          .withBrokerConfig(
+              cfg -> {
+                cfg.getExporters()
+                    .computeIfAbsent("elasticsearch", __ -> new ExporterCfg())
+                    .setArgs(
+                        Map.of(
+                            "index",
+                            Map.of("createTemplate", false),
+                            "retention",
+                            Map.of("enabled", false, "managePolicy", false),
+                            "authentication",
+                            Map.of("username", APP_USER, "password", APP_PASSWORD)));
+              })
+          .withOperateConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: ChrisKujawa
body: ❓ Will the archiver actually fail to start? 🤔 As it has no permission to create ILM policy?
created_at: 2025-03-10T13:58:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
+          // Enable security features
+          .withEnv("xpack.security.enabled", "true")
+          // Ensure a fast and reliable startup
+          .withStartupTimeout(Duration.ofMinutes(5))
+          .withStartupAttempts(3)
+          .withEnv("xpack.security.enabled", "true")
+          .withEnv("xpack.watcher.enabled", "false")
+          .withEnv("xpack.ml.enabled", "false")
+          // Configure with allowed repository storage path
+          .withEnv("path.repo", "~/")
+          // to be able to delete indices with wildcards
+          .withEnv("action.destructive_requires_name", "false")
+          .withCopyToContainer(
+              Transferable.of(APP_ROLE_DEFINITION), "/usr/share/elasticsearch/config/roles.yml")
+          .withClasspathResourceMapping(
+              "elasticsearch-fast-startup.options",
+              "/usr/share/elasticsearch/config/jvm.options.d/elasticsearch-fast-startup.options",
+              BindMode.READ_ONLY);
+
+  // Configure the Camunda single application with restricted access to the Elasticsearch
+  @TestZeebe(autoStart = false)
+  final TestStandaloneCamunda camunda =
+      new TestStandaloneCamunda(es, false)
+          .withAdditionalProfile(Profile.DEFAULT_AUTH_PROFILE)
+          .withProperty("camunda.operate.migration.migrationEnabled", false)
+          .withProperty("camunda.tasklist.migration.migrationEnabled", false)
+          .withProperty("camunda.database.username", APP_USER)
+          .withProperty("camunda.database.password", APP_PASSWORD)
+          .withBrokerConfig(
+              cfg -> {
+                cfg.getExporters()
+                    .computeIfAbsent("elasticsearch", __ -> new ExporterCfg())
+                    .setArgs(
+                        Map.of(
+                            "index",
+                            Map.of("createTemplate", false),
+                            "retention",
+                            Map.of("enabled", false, "managePolicy", false),
+                            "authentication",
+                            Map.of("username", APP_USER, "password", APP_PASSWORD)));
+              })
+          .withOperateConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
+              })
+          .withTasklistConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
+                cfg.getArchiver().setIlmManagePolicy(false);
+              });
+
+  // Elasticsearch client with admin access to apply some manual operations
+  private ElasticsearchClient adminElasticsearchClient;
+
+  @BeforeEach
+  void setup() throws IOException, InterruptedException {
+    // setup ES users
+    es.execInContainer("elasticsearch-users", "useradd", APP_USER, "-p", APP_PASSWORD);
+    es.execInContainer("elasticsearch-users", "useradd", ADMIN_USER, "-p", ADMIN_PASSWORD);
+    es.execInContainer("elasticsearch-users", "roles", ADMIN_USER, "-a", ADMIN_ROLE);
+    es.execInContainer("elasticsearch-users", "roles", APP_USER, "-a", APP_ROLE);
+    // Connect to ES
+    schemaManager.withProperty(
+        "zeebe.broker.exporters.elasticsearch.args.url", "http://" + es.getHttpHostAddress());
+    schemaManager.withProperty(
+        "camunda.operate.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    schemaManager.withProperty(
+        "camunda.tasklist.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    backupManager.withProperty(
+        "camunda.operate.elasticsearch.url", "http://" + es.getHttpHostAddress());
+    backupManager.withProperty(
+        "camunda.tasklist.elasticsearch.url", "http://" + es.getHttpHostAddress());
+
+    adminElasticsearchClient = createAdminElasticsearchClient("http://" + es.getHttpHostAddress());
+  }
+
+  @Test
+  void canBackupRestore() throws Exception {
+    // GIVEN
+    // create the schema
+    schemaManager.start();
+    // Create a snapshot repository to store backups
+    createSnapshotRepository();
+    // Start the Camunda application
+    camunda.start();
+    // Generate test data
+    final long processInstanceKey = generateData();
+    // Assert that the data is created in Elasticsearch
+    final long userTaskKey =
+        assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance());
+
+    // WHEN
+    // Start the backup process with a specific backup ID
+    backupManager.withBackupId(BACKUP_ID).start();
+    // manually backup zeebe indices
+    backupZeebeIndices();
+
+    final List<String> snapshots = new ArrayList<>();
+    // Wait for snapshots to be completed
+    snapshots.addAll(waitForSnapshotsToBeCompleted(OPERATE_SNAPSHOT_NAME_PREFIX, 6));
+    snapshots.addAll(waitForSnapshotsToBeCompleted(TASKLIST_SNAPSHOT_NAME_PREFIX, 6));
+    snapshots.addAll(waitForSnapshotsToBeCompleted(ZEEBE_SNAPSHOT_NAME, 1));
+    // Update the current state by completing the user task and the process instance
+    completeUserTask(userTaskKey);
+    // Assert that the state is updated: process instance is completed
+    assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance().negate());
+
+    // Stop the Camunda application
+    camunda.stop();
+    // Delete indices to simulate a fresh start
+    deleteIndices();
+    // Restore the backup to recover deleted data
+    restoreBackup(snapshots);
+    // Restart Camunda after restoration
+    camunda.start();
+
+    // THEN
+    // Validate that the data is restored successfully at the state before the backup: process
+    // instance is running
+    assertThatDataIsPresent(processInstanceKey, isRunningProcessInstance());
+  }
+
+  /**
+   * As documented in
+   * https://docs.camunda.io/docs/self-managed/operational-guides/backup-restore/backup-and-restore/#backup-process
+   */
+  private void backupZeebeIndices() throws IOException {
+    adminElasticsearchClient
+        .snapshot()
+        .create(
+            r ->
+                r.repository(REPOSITORY_NAME)
+                    .snapshot(ZEEBE_SNAPSHOT_NAME)
+                    .indices("zeebe-record*")
+                    .featureStates("none"));
+  }
+
+  private long generateData() {
+    // creating a user for webapps authentication
+    try (final var operateClient = camunda.newOperateClient()) {
+      operateClient.createUser(TEST_USER_NAME, TEST_USER_PASSWORD);
+    }
+    // creating a process instance with user task
+    final long processInstanceKey;
+    try (final var zeebeClient = camunda.newClientBuilder().build()) {
+      // Deploy process with user task
+      zeebeClient
+          .newDeployResourceCommand()
+          .addProcessModel(
+              Bpmn.createExecutableProcess("process-with-user-task")
+                  .startEvent()
+                  .userTask("user-task")
+                  .zeebeUserTask()
+                  .zeebeAssignee(TEST_USER_NAME)
+                  .endEvent()
+                  .done(),
+              "process-with-user-task.bpmn")
+          .send()
+          .join();
+      processInstanceKey =
+          zeebeClient
+              .newCreateInstanceCommand()
+              .bpmnProcessId("process-with-user-task")
+              .latestVersion()
+              .send()
+              .join()
+              .getProcessInstanceKey();
+    }
+    return processInstanceKey;
+  }
+
+  private List<String> waitForSnapshotsToBeCompleted(
+      final String snapshotNamePrefix, final int count) {
+    final var snapshots = new ArrayList<String>();
+    Awaitility.await("should find all snapshots completed")
+        .atMost(ofSeconds(30))
+        .pollDelay(ofSeconds(2))
+        .untilAsserted(
+            () -> {
+              final var response =
+                  adminElasticsearchClient
+                      .snapshot()
+                      .get(r -> r.repository(REPOSITORY_NAME).snapshot(snapshotNamePrefix + "*"))
+                      .snapshots();
+              assertThat(response).hasSize(count);
+              assertThat(response.stream().map(snapshot -> snapshot.state()))
+                  .allMatch(state -> "SUCCESS".equals(state));
+              snapshots.addAll(response.stream().map(snapshot -> snapshot.snapshot()).toList());
+            });
+    return snapshots;
+  }
+
+  /**
+   * Asserts that the data is present in Operate and Tasklist.
+   *
+   * <ul>
+   *   <li>Asserts that the process instance is present in Operate and verifies the check.
+   *   <li>Asserts that one user task is present in Tasklist and returns its key.
+   * </ul>
+   */
+  private long assertThatDataIsPresent(
+      final long processInstanceKey, final Predicate<ProcessInstance> processInstanceCheck) {
+    try (final var operateClient = camunda.newOperateClient()) {
+      Awaitility.await("should find a process instance")
+          .atMost(ofSeconds(30))
+          .ignoreExceptions()
+          .untilAsserted(
+              () -> {
+                final var result =
+                    operateClient
+                        .withAuthentication(TEST_USER_NAME, TEST_USER_PASSWORD)
+                        .getProcessInstanceWith(processInstanceKey);
+                EitherAssert.assertThat(result)
+                    .right()
+                    .isNotNull()
+                    .extracting(ProcessInstanceResult::total)
+                    .isEqualTo(1L);
+                final var processInstance = result.get().processInstances().getFirst();
+                assertThat(processInstance).matches(processInstanceCheck);
+              });
+    }
+    final var userTaskKey = new AtomicReference<Long>();
+    try (final var tasklistClient = camunda.newTasklistClient()) {
+      Awaitility.await("should find a user task")
+          .atMost(ofSeconds(30))
+          .ignoreExceptions()
+          .untilAsserted(
+              () -> {
+                final List<SearchQueryHit<TaskEntity>> hits =
+                    tasklistClient
+                        .withAuthentication(TEST_USER_NAME, TEST_USER_PASSWORD)
+                        .searchUserTasks(SearchQueryBuilders.query().build())
+                        .hits();
+                assertThat(hits).hasSize(1);
+                userTaskKey.set(hits.get(0).source().getKey());
+              });
+    }
+    return userTaskKey.get();
+  }
+
+  private void deleteIndices() throws IOException {
+    adminElasticsearchClient
+        .indices()
+        .delete(r -> r.index("operate*", "tasklist*", "zeebe*").ignoreUnavailable(true));
+    Awaitility.await("should delete indices")
+        .atMost(ofSeconds(30))
+        .pollDelay(ofSeconds(2))
+        .untilAsserted(
+            () ->
+                assertThat(
+                        adminElasticsearchClient
+                            .indices()
+                            .exists(
+                                r ->
+                                    r.index("operate*", "tasklist*", "zeebe*")
+                                        .allowNoIndices(false))
+                            .value())
+                    .isFalse());
+  }
+
+  private void restoreBackup(final List<String> snapshots) {
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: ChrisKujawa
body: 🔧 We could link here also to the docs https://docs.camunda.io/docs/self-managed/operational-guides/backup-restore/backup-and-restore/#restore
created_at: 2025-03-10T14:03:51Z
author_association: MEMBER
diff_hunk: @@ -122,8 +105,7 @@ public static void main(final String[] args) throws Exception {
   public void run(final String... args) throws Exception {
     if (args.length != 1) {
       throw new IllegalArgumentException(
-          String.format(
-              "Expected one argument, the backup ID, but got: [%s]", Arrays.asList(args)));
+          String.format("Expected one argument, the backup ID, but got %d", args.length));
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: ChrisKujawa
body: 🔧 Hm not sure I think would be interesting to see the arguments.
created_at: 2025-03-10T14:04:54Z
author_association: MEMBER
diff_hunk: @@ -109,8 +94,6 @@ public static void main(final String[] args) throws Exception {
             io.camunda.tasklist.JacksonConfig.class,
             io.camunda.operate.JacksonConfig.class)
         .addCommandLineProperties(true)
-        //        .properties(defaultActiveProfiles)
-        .listeners(new ApplicationErrorListener())
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: ChrisKujawa
body: ❓ We don't need it?
created_at: 2025-03-10T14:05:03Z
author_association: MEMBER
diff_hunk: @@ -155,46 +137,73 @@ public void run(final String... args) throws Exception {
       LOG.info(
           "Triggered ES snapshots for Tasklist indices: {}",
           tasklistBackupResponse.getScheduledSnapshots());
-
-      LOG.info("Will observe snapshot creation...");
-      boolean snapshotObservation = true;
-      do {
-        // we need to sleep here already otherwise the API fails, as it can't
-        // handle empty responses...
-        Thread.sleep(5 * 1_000);
-        final var operateBackup = operateBackupManager.getBackupState(backupId);
-        final var operateBackupState = operateBackup.getState();
-        final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
-        final var tasklistBackupState = tasklistBackup.getState();
-
-        LOG.info("Snapshot observation:");
-        LOG.info(
-            "Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
-        LOG.info(
-            "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
-
-        if (tasklistBackupState != BackupStateDto.IN_PROGRESS
-            && operateBackupState
-                != io.camunda.operate.webapp.management.dto.BackupStateDto.IN_PROGRESS) {
-          snapshotObservation = false;
-        }
-      } while (snapshotObservation);
-
     } catch (final Exception e) {
       LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
       throw e;
     }
 
-    LOG.info("... triggered Elasticsearch snapshot based on Camunda's backup procedure");
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        LOG.error("Backup with id:[{}] failed.{}", backupId, failureReason);
+        throw new IllegalStateException("Backup with id:[%d] failed".formatted(backupId));
+      }
+    }
+
+    LOG.info("Backup with id:[{}}] is completed!", backupId);
+  }
+
+  private boolean isCompletedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return operateBackupState == io.camunda.operate.webapp.management.dto.BackupStateDto.COMPLETED
+        && tasklistBackupState == BackupStateDto.COMPLETED;
+  }
+
+  private boolean isFailedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return EnumSet.of(
+                io.camunda.operate.webapp.management.dto.BackupStateDto.FAILED,
+                io.camunda.operate.webapp.management.dto.BackupStateDto.INCOMPATIBLE)
+            .contains(operateBackupState)
+        || EnumSet.of(BackupStateDto.FAILED, BackupStateDto.INCOMPATIBLE)
+            .contains(tasklistBackupState);
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: ChrisKujawa
body: ❓ What about partial?
created_at: 2025-03-10T14:07:33Z
author_association: MEMBER
diff_hunk: @@ -155,46 +137,73 @@ public void run(final String... args) throws Exception {
       LOG.info(
           "Triggered ES snapshots for Tasklist indices: {}",
           tasklistBackupResponse.getScheduledSnapshots());
-
-      LOG.info("Will observe snapshot creation...");
-      boolean snapshotObservation = true;
-      do {
-        // we need to sleep here already otherwise the API fails, as it can't
-        // handle empty responses...
-        Thread.sleep(5 * 1_000);
-        final var operateBackup = operateBackupManager.getBackupState(backupId);
-        final var operateBackupState = operateBackup.getState();
-        final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
-        final var tasklistBackupState = tasklistBackup.getState();
-
-        LOG.info("Snapshot observation:");
-        LOG.info(
-            "Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
-        LOG.info(
-            "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
-
-        if (tasklistBackupState != BackupStateDto.IN_PROGRESS
-            && operateBackupState
-                != io.camunda.operate.webapp.management.dto.BackupStateDto.IN_PROGRESS) {
-          snapshotObservation = false;
-        }
-      } while (snapshotObservation);
-
     } catch (final Exception e) {
       LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
       throw e;
     }
 
-    LOG.info("... triggered Elasticsearch snapshot based on Camunda's backup procedure");
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        LOG.error("Backup with id:[{}] failed.{}", backupId, failureReason);
+        throw new IllegalStateException("Backup with id:[%d] failed".formatted(backupId));
+      }
+    }
+
+    LOG.info("Backup with id:[{}}] is completed!", backupId);
+  }
+
+  private boolean isCompletedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return operateBackupState == io.camunda.operate.webapp.management.dto.BackupStateDto.COMPLETED
+        && tasklistBackupState == BackupStateDto.COMPLETED;
+  }
+
+  private boolean isFailedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return EnumSet.of(
+                io.camunda.operate.webapp.management.dto.BackupStateDto.FAILED,
+                io.camunda.operate.webapp.management.dto.BackupStateDto.INCOMPATIBLE)
+            .contains(operateBackupState)
+        || EnumSet.of(BackupStateDto.FAILED, BackupStateDto.INCOMPATIBLE)
+            .contains(tasklistBackupState);
   }
 
   @SpringBootConfiguration
   @EnableConfigurationProperties(BrokerBasedProperties.class)
   @ConfigurationPropertiesScan
   @ComponentScan(
-      // LEARNING - basePackageClasses - defining a class causes to scan the whole package with *
-      // doing
-      // a traversal to include all childs
+      // CAUTION: basePackageClasses makes Spring scan the entire package of the specified
+      // classes, including sub-packages.
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: ChrisKujawa
body: 👍🏼 
created_at: 2025-03-10T14:07:47Z
author_association: MEMBER
diff_hunk: @@ -155,46 +137,73 @@ public void run(final String... args) throws Exception {
       LOG.info(
           "Triggered ES snapshots for Tasklist indices: {}",
           tasklistBackupResponse.getScheduledSnapshots());
-
-      LOG.info("Will observe snapshot creation...");
-      boolean snapshotObservation = true;
-      do {
-        // we need to sleep here already otherwise the API fails, as it can't
-        // handle empty responses...
-        Thread.sleep(5 * 1_000);
-        final var operateBackup = operateBackupManager.getBackupState(backupId);
-        final var operateBackupState = operateBackup.getState();
-        final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
-        final var tasklistBackupState = tasklistBackup.getState();
-
-        LOG.info("Snapshot observation:");
-        LOG.info(
-            "Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
-        LOG.info(
-            "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
-
-        if (tasklistBackupState != BackupStateDto.IN_PROGRESS
-            && operateBackupState
-                != io.camunda.operate.webapp.management.dto.BackupStateDto.IN_PROGRESS) {
-          snapshotObservation = false;
-        }
-      } while (snapshotObservation);
-
     } catch (final Exception e) {
       LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
       throw e;
     }
 
-    LOG.info("... triggered Elasticsearch snapshot based on Camunda's backup procedure");
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        LOG.error("Backup with id:[{}] failed.{}", backupId, failureReason);
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: ChrisKujawa
body: ❓ Does the failure reason also contain the state? If not we should log it as well.

🔧 As we throw an exception, the error message can probably also just be inside the exception. Meaning no extra log necessary.
created_at: 2025-03-10T14:08:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: houssain-barouni
body: I am not sure if this is really needed, I think we can remove it, it is used by the healthcheck actuator which is not relevant to the backup manager. I will delete it
created_at: 2025-03-10T14:12:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.backup.repositoryName", "els-test")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.backup.repositoryName", "els-test");
+
+  // Configure the schema manager to create indices and templates in test setup
+  @TestZeebe(autoStart = false)
+  final TestStandaloneSchemaManager schemaManager =
+      new TestStandaloneSchemaManager()
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.className",
+              "io.camunda.zeebe.exporter.ElasticsearchExporter")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.index.createTemplate", "true")
+          .withProperty("zeebe.broker.exporters.elasticsearch.args.retention.enabled", "true")
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.username", ADMIN_USER)
+          .withProperty(
+              "zeebe.broker.exporters.elasticsearch.args.authentication.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.operate.archiver.ilmEnabled", "true")
+          .withProperty("camunda.tasklist.database", "elasticsearch")
+          .withProperty("camunda.tasklist.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.tasklist.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.tasklist.elasticsearch.healthCheckEnabled", "false")
+          .withProperty("camunda.tasklist.archiver.ilmEnabled", "true");
+
+  @Container
+  private final ElasticsearchContainer es =
+      new ElasticsearchContainer(
+              DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch")
+                  .withTag(RestClient.class.getPackage().getImplementationVersion()))
+          // Enable security features
+          .withEnv("xpack.security.enabled", "true")
+          // Ensure a fast and reliable startup
+          .withStartupTimeout(Duration.ofMinutes(5))
+          .withStartupAttempts(3)
+          .withEnv("xpack.security.enabled", "true")
+          .withEnv("xpack.watcher.enabled", "false")
+          .withEnv("xpack.ml.enabled", "false")
+          // Configure with allowed repository storage path
+          .withEnv("path.repo", "~/")
+          // to be able to delete indices with wildcards
+          .withEnv("action.destructive_requires_name", "false")
+          .withCopyToContainer(
+              Transferable.of(APP_ROLE_DEFINITION), "/usr/share/elasticsearch/config/roles.yml")
+          .withClasspathResourceMapping(
+              "elasticsearch-fast-startup.options",
+              "/usr/share/elasticsearch/config/jvm.options.d/elasticsearch-fast-startup.options",
+              BindMode.READ_ONLY);
+
+  // Configure the Camunda single application with restricted access to the Elasticsearch
+  @TestZeebe(autoStart = false)
+  final TestStandaloneCamunda camunda =
+      new TestStandaloneCamunda(es, false)
+          .withAdditionalProfile(Profile.DEFAULT_AUTH_PROFILE)
+          .withProperty("camunda.operate.migration.migrationEnabled", false)
+          .withProperty("camunda.tasklist.migration.migrationEnabled", false)
+          .withProperty("camunda.database.username", APP_USER)
+          .withProperty("camunda.database.password", APP_PASSWORD)
+          .withBrokerConfig(
+              cfg -> {
+                cfg.getExporters()
+                    .computeIfAbsent("elasticsearch", __ -> new ExporterCfg())
+                    .setArgs(
+                        Map.of(
+                            "index",
+                            Map.of("createTemplate", false),
+                            "retention",
+                            Map.of("enabled", false, "managePolicy", false),
+                            "authentication",
+                            Map.of("username", APP_USER, "password", APP_PASSWORD)));
+              })
+          .withOperateConfig(
+              cfg -> {
+                cfg.getElasticsearch().setCreateSchema(false);
+                cfg.getElasticsearch().setHealthCheckEnabled(false);
+                cfg.getElasticsearch().setUsername(APP_USER);
+                cfg.getElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getZeebeElasticsearch().setUsername(APP_USER);
+                cfg.getZeebeElasticsearch().setPassword(APP_PASSWORD);
+                cfg.getArchiver().setIlmEnabled(true);
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: houssain-barouni
body: the ILM is created by the schema manager app beforehand
created_at: 2025-03-10T14:13:48Z
author_association: COLLABORATOR
diff_hunk: @@ -122,8 +105,7 @@ public static void main(final String[] args) throws Exception {
   public void run(final String... args) throws Exception {
     if (args.length != 1) {
       throw new IllegalArgumentException(
-          String.format(
-              "Expected one argument, the backup ID, but got: [%s]", Arrays.asList(args)));
+          String.format("Expected one argument, the backup ID, but got %d", args.length));
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: houssain-barouni
body: before it was logging `Expected one argument, the backup ID, but got: [[]]` when missing the mandatory arg (99% of the cases) 
I will put it back without the brackets 
created_at: 2025-03-10T14:16:07Z
author_association: COLLABORATOR
diff_hunk: @@ -109,8 +94,6 @@ public static void main(final String[] args) throws Exception {
             io.camunda.tasklist.JacksonConfig.class,
             io.camunda.operate.JacksonConfig.class)
         .addCommandLineProperties(true)
-        //        .properties(defaultActiveProfiles)
-        .listeners(new ApplicationErrorListener())
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: houssain-barouni
body: `ApplicationErrorListener` catches all the failure events and transforms to `System.exit(-1);` which is translated to exit code 255 (since -1 is outside the range 0-254)
created_at: 2025-03-10T14:17:33Z
author_association: COLLABORATOR
diff_hunk: @@ -155,46 +137,73 @@ public void run(final String... args) throws Exception {
       LOG.info(
           "Triggered ES snapshots for Tasklist indices: {}",
           tasklistBackupResponse.getScheduledSnapshots());
-
-      LOG.info("Will observe snapshot creation...");
-      boolean snapshotObservation = true;
-      do {
-        // we need to sleep here already otherwise the API fails, as it can't
-        // handle empty responses...
-        Thread.sleep(5 * 1_000);
-        final var operateBackup = operateBackupManager.getBackupState(backupId);
-        final var operateBackupState = operateBackup.getState();
-        final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
-        final var tasklistBackupState = tasklistBackup.getState();
-
-        LOG.info("Snapshot observation:");
-        LOG.info(
-            "Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
-        LOG.info(
-            "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
-
-        if (tasklistBackupState != BackupStateDto.IN_PROGRESS
-            && operateBackupState
-                != io.camunda.operate.webapp.management.dto.BackupStateDto.IN_PROGRESS) {
-          snapshotObservation = false;
-        }
-      } while (snapshotObservation);
-
     } catch (final Exception e) {
       LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
       throw e;
     }
 
-    LOG.info("... triggered Elasticsearch snapshot based on Camunda's backup procedure");
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        LOG.error("Backup with id:[{}] failed.{}", backupId, failureReason);
+        throw new IllegalStateException("Backup with id:[%d] failed".formatted(backupId));
+      }
+    }
+
+    LOG.info("Backup with id:[{}}] is completed!", backupId);
+  }
+
+  private boolean isCompletedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return operateBackupState == io.camunda.operate.webapp.management.dto.BackupStateDto.COMPLETED
+        && tasklistBackupState == BackupStateDto.COMPLETED;
+  }
+
+  private boolean isFailedBackup(
+      final io.camunda.operate.webapp.management.dto.BackupStateDto operateBackupState,
+      final BackupStateDto tasklistBackupState) {
+    return EnumSet.of(
+                io.camunda.operate.webapp.management.dto.BackupStateDto.FAILED,
+                io.camunda.operate.webapp.management.dto.BackupStateDto.INCOMPATIBLE)
+            .contains(operateBackupState)
+        || EnumSet.of(BackupStateDto.FAILED, BackupStateDto.INCOMPATIBLE)
+            .contains(tasklistBackupState);
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: houssain-barouni
body: PARTIAL is not part of the global status computed inside the services
the PARTIAL state is mapped to global FAILED if any
created_at: 2025-03-10T14:18:18Z
author_association: COLLABORATOR
diff_hunk: @@ -155,46 +137,73 @@ public void run(final String... args) throws Exception {
       LOG.info(
           "Triggered ES snapshots for Tasklist indices: {}",
           tasklistBackupResponse.getScheduledSnapshots());
-
-      LOG.info("Will observe snapshot creation...");
-      boolean snapshotObservation = true;
-      do {
-        // we need to sleep here already otherwise the API fails, as it can't
-        // handle empty responses...
-        Thread.sleep(5 * 1_000);
-        final var operateBackup = operateBackupManager.getBackupState(backupId);
-        final var operateBackupState = operateBackup.getState();
-        final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
-        final var tasklistBackupState = tasklistBackup.getState();
-
-        LOG.info("Snapshot observation:");
-        LOG.info(
-            "Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
-        LOG.info(
-            "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
-
-        if (tasklistBackupState != BackupStateDto.IN_PROGRESS
-            && operateBackupState
-                != io.camunda.operate.webapp.management.dto.BackupStateDto.IN_PROGRESS) {
-          snapshotObservation = false;
-        }
-      } while (snapshotObservation);
-
     } catch (final Exception e) {
       LOG.error("Expected to trigger ES snapshots for backupId {}, but failed", backupId, e);
       throw e;
     }
 
-    LOG.info("... triggered Elasticsearch snapshot based on Camunda's backup procedure");
+    LOG.info("Will observe snapshot creation...");
+    boolean isBackupInProgress = true;
+    while (isBackupInProgress) {
+      // we need to sleep here already otherwise the getBackupState fails, as it can't
+      // handle empty snapshot responses...
+      Thread.sleep(5 * 1_000);
+      final var operateBackup = operateBackupManager.getBackupState(backupId);
+      final var operateBackupState = operateBackup.getState();
+      final var tasklistBackup = tasklistBackupManager.getBackupState(backupId);
+      final var tasklistBackupState = tasklistBackup.getState();
+
+      LOG.info("Snapshot observation:");
+      LOG.info("Operate indices snapshot is {}. Details: [{}]", operateBackupState, operateBackup);
+      LOG.info(
+          "Tasklist indices snapshot is {}. Details: [{}]", tasklistBackupState, tasklistBackup);
+
+      if (isCompletedBackup(operateBackupState, tasklistBackupState)) {
+        isBackupInProgress = false;
+      } else if (isFailedBackup(operateBackupState, tasklistBackupState)) {
+        final var failureReason = new StringBuilder();
+        if (operateBackup.getFailureReason() != null
+            && !operateBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Operate indices backup failure: %s.".formatted(operateBackup.getFailureReason()));
+        }
+        if (tasklistBackup.getFailureReason() != null
+            && !tasklistBackup.getFailureReason().isEmpty()) {
+          failureReason.append(
+              " Tasklist indices backup failure: %s.".formatted(tasklistBackup.getFailureReason()));
+        }
+        LOG.error("Backup with id:[{}] failed.{}", backupId, failureReason);
path: dist/src/main/java/io/camunda/application/StandaloneBackupManager.java
user: login: houssain-barouni
body: > Does the failure reason also contain the state? If not we should log it as well.

yes, it is a summary of the failure cause and which snapshots are impacted and built this way https://github.com/camunda/camunda/blob/925c229cc6c5e31b4091d5484d14cb40cba1d6f5/operate/webapp/src/main/java/io/camunda/operate/webapp/elasticsearch/backup/ElasticsearchBackupRepository.java#L414-L435

>🔧 As we throw an exception, the error message can probably also just be inside the exception. Meaning no extra log necessary.

👍 
created_at: 2025-03-10T14:27:29Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,446 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.it;
+
+import static java.time.Duration.ofSeconds;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import co.elastic.clients.elasticsearch.ElasticsearchClient;
+import io.camunda.application.Profile;
+import io.camunda.operate.webapp.api.v1.entities.ProcessInstance;
+import io.camunda.operate.webapp.backup.Metadata;
+import io.camunda.qa.util.cluster.TestRestOperateClient.ProcessInstanceResult;
+import io.camunda.qa.util.cluster.TestStandaloneBackupManager;
+import io.camunda.qa.util.cluster.TestStandaloneCamunda;
+import io.camunda.qa.util.cluster.TestStandaloneSchemaManager;
+import io.camunda.search.clients.core.SearchQueryHit;
+import io.camunda.search.clients.query.SearchQueryBuilders;
+import io.camunda.search.connect.configuration.ConnectConfiguration;
+import io.camunda.search.connect.es.ElasticsearchConnector;
+import io.camunda.tasklist.entities.TaskEntity;
+import io.camunda.zeebe.broker.system.configuration.ExporterCfg;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration;
+import io.camunda.zeebe.qa.util.junit.ZeebeIntegration.TestZeebe;
+import io.camunda.zeebe.test.util.asserts.EitherAssert;
+import java.io.IOException;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+import org.awaitility.Awaitility;
+import org.elasticsearch.client.RestClient;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.BindMode;
+import org.testcontainers.elasticsearch.ElasticsearchContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+
+@ZeebeIntegration
+@Testcontainers
+final class StandaloneBackupManagerTest {
+
+  public static final String ADMIN_USER = "camunda-admin";
+  public static final String ADMIN_PASSWORD = "admin123";
+  public static final String ADMIN_ROLE = "superuser";
+
+  public static final String APP_USER = "camunda-app";
+  public static final String APP_PASSWORD = "app123";
+  public static final String APP_ROLE = "camunda_app_role";
+  public static final String APP_ROLE_DEFINITION =
+      // language=yaml
+      """
+      camunda_app_role:
+        indices:
+          - names: ['zeebe-*', 'operate-*', 'tasklist-*']
+            privileges: ['manage', 'read', 'write']
+      """;
+
+  private static final String TEST_USER_NAME = "foo";
+  private static final String TEST_USER_PASSWORD = "bar";
+
+  private static final String REPOSITORY_NAME = "els-test";
+
+  private static final long BACKUP_ID = 12345L;
+  public static final String ZEEBE_SNAPSHOT_NAME = "camunda_zeebe_records_backup_" + BACKUP_ID;
+  public static final String TASKLIST_SNAPSHOT_NAME_PREFIX =
+      io.camunda.tasklist.webapp.es.backup.Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+  public static final String OPERATE_SNAPSHOT_NAME_PREFIX =
+      Metadata.buildSnapshotNamePrefix(BACKUP_ID);
+
+  // Configure the backup manager for testing
+  @TestZeebe(autoStart = false)
+  final TestStandaloneBackupManager backupManager =
+      new TestStandaloneBackupManager()
+          .withProperty("camunda.operate.elasticsearch.username", ADMIN_USER)
+          .withProperty("camunda.operate.elasticsearch.password", ADMIN_PASSWORD)
+          .withProperty("camunda.operate.elasticsearch.healthCheckEnabled", "false")
path: qa/integration-tests/src/test/java/io/camunda/it/StandaloneBackupManagerTest.java
user: login: houssain-barouni
body: I just figured out that it is required because the health check implementation in Tasklist and Operate requires cluster privileges, and this is invoked at application startup
created_at: 2025-03-10T15:46:54Z
author_association: COLLABORATOR

PR comments:
user: login: ChrisKujawa
created_at: 2025-03-10T20:39:26Z
body: /backport
author_association: MEMBER
user: login: backport-action
created_at: 2025-03-10T20:39:57Z
body: Successfully created backport PR for `stable/8.7`:
- #29408
author_association: COLLABORATOR