Repository: camunda/camunda
Pull request #29804: feat: add basic command distribution metrics
Author: login: eppdot
Labels: name: component/zeebe
description: Related to the Zeebe component/team
Created: 2025-03-19T08:00:00Z | Merged: 2025-05-26T12:02:02Z

## Description

This PR focusses on the basic command distribution metrics, like:
  - current number of command distributions
  - current number of pending distributions per partition
  - current number of inflight distributions per partition
  - number of overall distributions
  - number of retries for distributions per partition
  - number of acknowledgements sent/received by target partition

**TODO**
  - [x] implement actual micrometer metrics
  - [x] implement proper reset behavior in case of recovery (to stay in sync with state/rocksdb)
  - [ ] evaluate Grafana to provide higher level metrics (e.g. age/latency of distributions)

My tendency is to bring this changes to production and then add more complex metrics if Grafana does not support this out-of-the-box.

Follow Up:
- [ ] Add timer metrics for command distribution latency
- [ ] Add long timer metric for redistribute scheduled task
- [ ] Make every distribution uses a (tmp) queue

## Checklist

<!--- Please delete options that are not relevant. Boxes should be checked by reviewer. -->
- [ ] for CI changes:
  - [ ] structural/foundational changes signed off by [CI DRI](https://github.com/cmur2)
  - [ ] [ci.yml](https://github.com/camunda/camunda/blob/main/.github/workflows/ci.yml) modifications comply with ["Unified CI" requirements](https://github.com/camunda/camunda/wiki/CI-&-Automation#workflow-inclusion-criteria)
  - [ ] enable backports [when recommended](https://github.com/camunda/camunda/wiki/CI-&-Automation#when-to-backport-ci-changes)

## Related issues

closes #32417


Linked issue #32417: Track metrics during command distribution in Zeebe

Diff:
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
new file mode 100644
index 000000000000..b0d37b5fa93a
--- /dev/null
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
@@ -0,0 +1,217 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.common.docs.KeyName;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import io.micrometer.core.instrument.Tags;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void reset() {
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void startedDistribution() {
+    distributionsCounter.increment();
+  }
+
+  public void addActiveDistribution() {
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeActiveDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).removePendingDistribution();
+  }
+
+  public void addInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addInflightDistribution();
+  }
+
+  public void removeInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).removeInflightDistribution();
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed and an acknowledgement is sent back to the origin partition.
+   *
+   * @param originPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   */
+  public void sentAcknowledgeDistribution(final int originPartitionId) {
+    getPartitionMetrics(originPartitionId).sentAcknowledgeDistribution();
+  }
+
+  /**
+   * This method is called on the origin partition when a distribution command has been successfully
+   * acknowledged by the target partition.
+   *
+   * @param targetPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   */
+  public void receivedAcknowledgeDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).receivedAcknowledgeDistribution();
+  }
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See {@link io.camunda.zeebe.engine.processing.distribution.CommandRedistributor}.
+   *
+   * @param targetPartitionId the target partition id of the distribution
+   */
+  public void retryInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).retryInflightDistribution();
+  }
+
+  private PartitionDistributionMetrics getPartitionMetrics(final int targetPartitionId) {
+    return partitionMetrics.computeIfAbsent(
+        targetPartitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  private static class PartitionDistributionMetrics {
+
+    private final MeterRegistry meterRegistry;
+    private final int targetPartitionId;
+
+    // Metrics
+    private final StatefulGauge pendingDistributionsGauge;
+    private final StatefulGauge inflightDistributionsGauge;
+    private final Counter retryInflightDistributionsCounter;
+    private final Counter receivedAcknowledgeDistributionsCounter;
+    private final Counter sentAcknowledgeDistributionsCounter;
+
+    public PartitionDistributionMetrics(
+        final int targetPartitionId, final MeterRegistry meterRegistry) {
+      this.targetPartitionId = targetPartitionId;
+      this.meterRegistry = meterRegistry;
+
+      pendingDistributionsGauge =
+          StatefulGauge.builder(DistributionMetricsDoc.PENDING_COMMAND_DISTRIBUTIONS.getName())
+              .description(DistributionMetricsDoc.PENDING_COMMAND_DISTRIBUTIONS.getDescription())
+              .tags(DistributionMetricsKeyNames.tags(targetPartitionId))
+              .register(meterRegistry);
+
+      inflightDistributionsGauge =
+          StatefulGauge.builder(DistributionMetricsDoc.INFLIGHT_COMMAND_DISTRIBUTIONS.getName())
+              .description(DistributionMetricsDoc.INFLIGHT_COMMAND_DISTRIBUTIONS.getDescription())
+              .tags(DistributionMetricsKeyNames.tags(targetPartitionId))
+              .register(meterRegistry);
+
+      retryInflightDistributionsCounter =
+          Counter.builder(DistributionMetricsDoc.RETRY_INFLIGHT_COMMAND_DISTRIBUTIONS.getName())
+              .description(
+                  DistributionMetricsDoc.RETRY_INFLIGHT_COMMAND_DISTRIBUTIONS.getDescription())
+              .tags(DistributionMetricsKeyNames.tags(targetPartitionId))
+              .register(meterRegistry);
+
+      sentAcknowledgeDistributionsCounter =
+          Counter.builder(DistributionMetricsDoc.SENT_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS.getName())
+              .description(
+                  DistributionMetricsDoc.SENT_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS.getDescription())
+              .tags(DistributionMetricsKeyNames.tags(targetPartitionId))
+              .register(meterRegistry);
+
+      receivedAcknowledgeDistributionsCounter =
+          Counter.builder(
+                  DistributionMetricsDoc.RECEIVED_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS.getName())
+              .description(
+                  DistributionMetricsDoc.RECEIVED_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS
+                      .getDescription())
+              .tags(DistributionMetricsKeyNames.tags(targetPartitionId))
+              .register(meterRegistry);
+    }
+
+    public void resetGauges() {
+      pendingDistributionsGauge.set(0);
+      inflightDistributionsGauge.set(0);
+    }
+
+    public void addPendingDistribution() {
+      pendingDistributionsGauge.increment();
+    }
+
+    public void removePendingDistribution() {
+      pendingDistributionsGauge.decrement();
+    }
+
+    public void addInflightDistribution() {
+      inflightDistributionsGauge.increment();
+    }
+
+    public void removeInflightDistribution() {
+      inflightDistributionsGauge.decrement();
+    }
+
+    public void retryInflightDistribution() {
+      retryInflightDistributionsCounter.increment();
+    }
+
+    public void sentAcknowledgeDistribution() {
+      sentAcknowledgeDistributionsCounter.increment();
+    }
+
+    public void receivedAcknowledgeDistribution() {
+      receivedAcknowledgeDistributionsCounter.increment();
+    }
+
+    public enum DistributionMetricsKeyNames implements KeyName {
+      /** The ID of the partition associated to the metric */
+      TARGET_PARTITION {
+        @Override
+        public String asString() {
+          return "targetPartition";
+        }
+      };
+
+      public static Tags tags(final int partitionId) {
+        return Tags.of(TARGET_PARTITION.asString(), String.valueOf(partitionId));
+      }
+    }
+  }
+}
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetricsDoc.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetricsDoc.java
new file mode 100644
index 000000000000..173bebf1f695
--- /dev/null
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetricsDoc.java
@@ -0,0 +1,177 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.ExtendedMeterDocumentation;
+import io.micrometer.common.docs.KeyName;
+import io.micrometer.core.instrument.Meter.Type;
+import io.micrometer.core.instrument.Tags;
+
+public enum DistributionMetricsDoc implements ExtendedMeterDocumentation {
+  COMMAND_DISTRIBUTIONS {
+    @Override
+    public String getDescription() {
+      return "Counts the number of command distributions.";
+    }
+
+    @Override
+    public String getName() {
+      return "zeebe.command.distributions";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.COUNTER;
+    }
+  },
+
+  ACTIVE_COMMAND_DISTRIBUTIONS {
+    @Override
+    public String getDescription() {
+      return "Tracks the number of currently active command distributions.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".active";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.GAUGE;
+    }
+  },
+
+  PENDING_COMMAND_DISTRIBUTIONS {
+
+    @Override
+    public String getDescription() {
+      return "Tracks the number of currently pending command distributions.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".pending";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.GAUGE;
+    }
+
+    @Override
+    public KeyName[] getAdditionalKeyNames() {
+      return DistributionMetricKeyNames.values();
+    }
+  },
+
+  INFLIGHT_COMMAND_DISTRIBUTIONS {
+
+    @Override
+    public String getDescription() {
+      return "Tracks the number of currently inflight command distributions.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".inflight";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.GAUGE;
+    }
+
+    @Override
+    public KeyName[] getAdditionalKeyNames() {
+      return DistributionMetricKeyNames.values();
+    }
+  },
+
+  RETRY_INFLIGHT_COMMAND_DISTRIBUTIONS {
+
+    @Override
+    public String getDescription() {
+      return "Counts the number of retried inflight command distributions.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".inflight.retries";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.COUNTER;
+    }
+
+    @Override
+    public KeyName[] getAdditionalKeyNames() {
+      return DistributionMetricKeyNames.values();
+    }
+  },
+
+  RECEIVED_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS {
+
+    @Override
+    public String getDescription() {
+      return "Counts the number of received acknowledgements from the target partition.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".acknowledged.received";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.COUNTER;
+    }
+
+    @Override
+    public KeyName[] getAdditionalKeyNames() {
+      return DistributionMetricKeyNames.values();
+    }
+  },
+
+  SENT_ACKNOWLEDGE_COMMAND_DISTRIBUTIONS {
+
+    @Override
+    public String getDescription() {
+      return "Counts the number of sent acknowledgements to the target partition.";
+    }
+
+    @Override
+    public String getName() {
+      return COMMAND_DISTRIBUTIONS.getName() + ".acknowledged.sent";
+    }
+
+    @Override
+    public Type getType() {
+      return Type.COUNTER;
+    }
+
+    @Override
+    public KeyName[] getAdditionalKeyNames() {
+      return DistributionMetricKeyNames.values();
+    }
+  };
+
+  public enum DistributionMetricKeyNames implements KeyName {
+    TARGET_PARTITION {
+      @Override
+      public String asString() {
+        return "targetPartition";
+      }
+    };
+
+    public static Tags targetPartitionTags(final int targetPartitionId) {
+      return Tags.of(TARGET_PARTITION.asString(), String.valueOf(targetPartitionId));
+    }
+  }
+}
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
index cb287e8c2274..d1fe738094a7 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
@@ -12,6 +12,7 @@
 import io.camunda.search.clients.SearchClientsProxy;
 import io.camunda.zeebe.dmn.DecisionEngineFactory;
 import io.camunda.zeebe.engine.EngineConfiguration;
+import io.camunda.zeebe.engine.metrics.DistributionMetrics;
 import io.camunda.zeebe.engine.metrics.JobProcessingMetrics;
 import io.camunda.zeebe.engine.metrics.ProcessEngineMetrics;
 import io.camunda.zeebe.engine.processing.batchoperation.BatchOperationSetupProcessors;
@@ -116,6 +117,8 @@ public static TypedRecordProcessors createEngineProcessors(
     final var jobMetrics = new JobProcessingMetrics(typedRecordProcessorContext.getMeterRegistry());
     final var processEngineMetrics =
         new ProcessEngineMetrics(typedRecordProcessorContext.getMeterRegistry());
+    final var distributionMetrics =
+        new DistributionMetrics(typedRecordProcessorContext.getMeterRegistry());
 
     subscriptionCommandSender.setWriters(writers);
 
@@ -145,7 +148,8 @@ public static TypedRecordProcessors createEngineProcessors(
             writers,
             typedRecordProcessorContext.getPartitionId(),
             routingInfo,
-            interPartitionCommandSender);
+            interPartitionCommandSender,
+            distributionMetrics);
 
     final var deploymentDistributionCommandSender =
         new DeploymentDistributionCommandSender(
@@ -236,11 +240,10 @@ public static TypedRecordProcessors createEngineProcessors(
         authCheckBehavior);
     addCommandDistributionProcessors(
         commandDistributionBehavior,
+        scheduledTaskStateFactory,
         typedRecordProcessors,
         writers,
-        processingState,
-        scheduledTaskStateFactory,
-        interPartitionCommandSender);
+        processingState);
 
     UserProcessors.addUserProcessors(
         keyGenerator,
@@ -580,16 +583,16 @@ private static void addUserTaskProcessors(
 
   private static void addCommandDistributionProcessors(
       final CommandDistributionBehavior commandDistributionBehavior,
+      final Supplier<ScheduledTaskState> scheduledTaskStateSupplier,
       final TypedRecordProcessors typedRecordProcessors,
       final Writers writers,
-      final ProcessingState processingState,
-      final Supplier<ScheduledTaskState> scheduledTaskStateFactory,
-      final InterPartitionCommandSender interPartitionCommandSender) {
+      final ProcessingState processingState) {
 
     // periodically retries command distribution
     typedRecordProcessors.withListener(
         new CommandRedistributor(
-            scheduledTaskStateFactory.get().getDistributionState(), interPartitionCommandSender));
+            commandDistributionBehavior.withScheduledState(
+                scheduledTaskStateSupplier.get().getDistributionState())));
 
     final var distributionState = processingState.getDistributionState();
     typedRecordProcessors.onCommand(
@@ -600,10 +603,12 @@ private static void addCommandDistributionProcessors(
     typedRecordProcessors.onCommand(
         ValueType.COMMAND_DISTRIBUTION,
         CommandDistributionIntent.FINISH,
-        new CommandDistributionFinishProcessor(writers, commandDistributionBehavior));
+        new CommandDistributionFinishProcessor(commandDistributionBehavior));
     typedRecordProcessors.onCommand(
         ValueType.COMMAND_DISTRIBUTION,
         CommandDistributionIntent.CONTINUE,
         new CommandDistributionContinueProcessor(distributionState, writers));
+
+    typedRecordProcessors.withListener(commandDistributionBehavior);
   }
 }
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
index 1108a00daae9..4f3c63acf593 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
@@ -9,14 +9,11 @@
 
 import io.camunda.zeebe.engine.processing.ExcludeAuthorizationCheck;
 import io.camunda.zeebe.engine.processing.streamprocessor.TypedRecordProcessor;
-import io.camunda.zeebe.engine.processing.streamprocessor.writers.StateWriter;
-import io.camunda.zeebe.engine.processing.streamprocessor.writers.TypedCommandWriter;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.TypedRejectionWriter;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.Writers;
 import io.camunda.zeebe.engine.state.immutable.DistributionState;
 import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
 import io.camunda.zeebe.protocol.record.RejectionType;
-import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
 import io.camunda.zeebe.stream.api.records.TypedRecord;
 
 @ExcludeAuthorizationCheck
@@ -30,8 +27,6 @@ public class CommandDistributionAcknowledgeProcessor
 
   private final CommandDistributionBehavior commandDistributionBehavior;
   private final DistributionState distributionState;
-  private final TypedCommandWriter commandWriter;
-  private final StateWriter stateWriter;
   private final TypedRejectionWriter rejectionWriter;
 
   public CommandDistributionAcknowledgeProcessor(
@@ -40,8 +35,6 @@ public CommandDistributionAcknowledgeProcessor(
       final Writers writers) {
     this.commandDistributionBehavior = commandDistributionBehavior;
     this.distributionState = distributionState;
-    commandWriter = writers.command();
-    stateWriter = writers.state();
     rejectionWriter = writers.rejection();
   }
 
@@ -51,6 +44,8 @@ public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
     final var recordValue = record.getValue();
     final var partitionId = recordValue.getPartitionId();
 
+    commandDistributionBehavior.getMetrics().receivedAcknowledgeDistribution(partitionId);
+
     if (!distributionState.hasPendingDistribution(distributionKey, partitionId)) {
       rejectionWriter.appendRejection(
           record,
@@ -59,23 +54,6 @@ public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
       return;
     }
 
-    stateWriter.appendFollowUpEvent(
-        distributionKey, CommandDistributionIntent.ACKNOWLEDGED, recordValue);
-
-    final var queueId = distributionState.getQueueIdForDistribution(distributionKey);
-    queueId.ifPresent(
-        queue -> commandDistributionBehavior.distributeNextInQueue(queue, partitionId));
-
-    if (!distributionState.hasPendingDistribution(distributionKey)) {
-      final var finishRecord =
-          new CommandDistributionRecord()
-              .setPartitionId(record.getPartitionId())
-              .setValueType(recordValue.getValueType())
-              .setIntent(recordValue.getIntent());
-      queueId.ifPresent(finishRecord::setQueueId);
-
-      commandWriter.appendFollowUpCommand(
-          distributionKey, CommandDistributionIntent.FINISH, finishRecord);
-    }
+    commandDistributionBehavior.onAcknowledgeDistribution(distributionKey, recordValue);
   }
 }
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
index cca1060601c2..092e5701e9ca 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
@@ -7,12 +7,14 @@
  */
 package io.camunda.zeebe.engine.processing.distribution;
 
+import io.camunda.zeebe.engine.metrics.DistributionMetrics;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.SideEffectWriter;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.StateWriter;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.TypedCommandWriter;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.Writers;
 import io.camunda.zeebe.engine.state.distribution.DistributionQueue;
 import io.camunda.zeebe.engine.state.immutable.DistributionState;
+import io.camunda.zeebe.engine.state.immutable.DistributionState.PendingDistributionVisitor;
 import io.camunda.zeebe.engine.state.routing.RoutingInfo;
 import io.camunda.zeebe.protocol.Protocol;
 import io.camunda.zeebe.protocol.impl.record.UnifiedRecordValue;
@@ -21,6 +23,8 @@
 import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
 import io.camunda.zeebe.protocol.record.intent.Intent;
 import io.camunda.zeebe.stream.api.InterPartitionCommandSender;
+import io.camunda.zeebe.stream.api.ReadonlyStreamProcessorContext;
+import io.camunda.zeebe.stream.api.StreamProcessorLifecycleAware;
 import io.camunda.zeebe.stream.api.records.TypedRecord;
 import java.util.Objects;
 import java.util.Optional;
@@ -38,7 +42,7 @@
  *     href="https://github.com/camunda/camunda/blob/main/zeebe/docs/generalized_distribution.md">
  *     generalized_distribution.md</a>
  */
-public final class CommandDistributionBehavior {
+public final class CommandDistributionBehavior implements StreamProcessorLifecycleAware {
 
   private final DistributionState distributionState;
   private final TypedCommandWriter commandWriter;
@@ -59,19 +63,61 @@ public final class CommandDistributionBehavior {
   private final CommandDistributionRecord commandDistributionContinuation =
       new CommandDistributionRecord();
 
+  private final DistributionMetrics distributionMetrics;
+  private final Writers writers;
+
   public CommandDistributionBehavior(
       final DistributionState distributionState,
       final Writers writers,
       final int currentPartition,
       final RoutingInfo routingInfo,
-      final InterPartitionCommandSender partitionCommandSender) {
+      final InterPartitionCommandSender partitionCommandSender,
+      final DistributionMetrics distributionMetrics) {
     this.distributionState = distributionState;
+    this.writers = writers;
     commandWriter = writers.command();
     stateWriter = writers.state();
     sideEffectWriter = writers.sideEffect();
     this.routingInfo = routingInfo;
     interPartitionCommandSender = partitionCommandSender;
     currentPartitionId = currentPartition;
+    this.distributionMetrics = distributionMetrics;
+  }
+
+  public CommandDistributionBehavior withScheduledState(final DistributionState state) {
+    return new CommandDistributionBehavior(
+        state,
+        writers,
+        currentPartitionId,
+        routingInfo,
+        interPartitionCommandSender,
+        distributionMetrics);
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    getMetrics().reset();
+
+    getDistributionState()
+        .foreachCommandDistribution(
+            (distributionKey, record) -> {
+              getMetrics().addActiveDistribution();
+              return true;
+            });
+
+    getDistributionState()
+        .foreachPendingDistribution(
+            (distributionKey, record) -> {
+              getMetrics().addPendingDistribution(record.getPartitionId());
+              return true;
+            });
+
+    getDistributionState()
+        .foreachRetriableDistribution(
+            (distributionKey, record) -> {
+              getMetrics().addInflightDistribution(record.getPartitionId());
+              return true;
+            });
   }
 
   /**
@@ -141,6 +187,9 @@ private <T extends UnifiedRecordValue> void distributeCommand(
           }
           distributeToPartition(partition, distributionRecord, distributionKey);
         });
+
+    getMetrics().startedDistribution();
+    getMetrics().addActiveDistribution();
   }
 
   private void distributeToPartition(
@@ -162,6 +211,8 @@ private void distributeToPartition(
     if (canDistributeImmediately) {
       startDistributing(partition, distributionRecord, distributionKey);
     }
+
+    getMetrics().addPendingDistribution(partition);
   }
 
   private void enqueueDistribution(
@@ -176,7 +227,7 @@ private void enqueueDistribution(
   /**
    * If the given distribution was part of a queue, the next distribution from the queue is started.
    */
-  void distributeNextInQueue(final String queue, final int partition) {
+  private void distributeNextInQueue(final String queue, final int partition) {
     distributionState
         .getNextQueuedDistributionKey(queue, partition)
         .ifPresent(
@@ -187,7 +238,7 @@ void distributeNextInQueue(final String queue, final int partition) {
                     nextDistributionKey));
   }
 
-  void continueAfterQueue(final String queue) {
+  private void continueAfterQueue(final String queue) {
     if (distributionState.hasQueuedDistributions(queue)) {
       return;
     }
@@ -209,6 +260,7 @@ private void startDistributing(
       final long distributionKey) {
     final var valueType = distributionRecord.getValueType();
     final var intent = distributionRecord.getIntent();
+    final var queueId = distributionRecord.getQueueId();
 
     commandDistributionDistributing.reset();
 
@@ -220,8 +272,11 @@ private void startDistributing(
         commandDistributionDistributing
             .setPartitionId(partition)
             .setValueType(valueType)
+            .setQueueId(queueId)
             .setIntent(intent));
 
+    getMetrics().addInflightDistribution(partition);
+
     // This getter makes a hard copy of the command value, which we need to send the command to the
     // other partition in a side effect. It does not appear to be possible to reuse a single
     // instance for distributing to all partitions in the form of a method parameter, but it's not
@@ -264,6 +319,8 @@ public <T extends UnifiedRecordValue> void acknowledgeCommand(final TypedRecord<
               acknowledgeRecord);
           return true;
         });
+
+    getMetrics().sentAcknowledgeDistribution(receiverPartitionId);
   }
 
   private <T extends UnifiedRecordValue> void requestContinuation(
@@ -290,6 +347,76 @@ private <T extends UnifiedRecordValue> void requestContinuation(
         key, CommandDistributionIntent.CONTINUATION_REQUESTED, commandDistributionContinuation);
   }
 
+  public DistributionMetrics getMetrics() {
+    return distributionMetrics;
+  }
+
+  public DistributionState getDistributionState() {
+    return distributionState;
+  }
+
+  public InterPartitionCommandSender getCommandSender() {
+    return interPartitionCommandSender;
+  }
+
+  public void onAcknowledgeDistribution(
+      final long distributionKey, final CommandDistributionRecord recordValue) {
+    final var partitionId = recordValue.getPartitionId();
+
+    // finish the distribution for the partition that acknowledged the distribution
+    stateWriter.appendFollowUpEvent(
+        distributionKey, CommandDistributionIntent.ACKNOWLEDGED, recordValue);
+
+    getMetrics().removeInflightDistribution(partitionId);
+    getMetrics().removePendingDistribution(partitionId);
+
+    // Distribute next in queue if the distribution was part of a queue
+    final var queueId = distributionState.getQueueIdForDistribution(distributionKey);
+    queueId.ifPresent(queue -> distributeNextInQueue(queue, partitionId));
+
+    // Finish the distribution if there are no more pending distributions
+    if (!distributionState.hasPendingDistribution(distributionKey)) {
+      final var finishRecord =
+          new CommandDistributionRecord()
+              .setPartitionId(currentPartitionId)
+              .setValueType(recordValue.getValueType())
+              .setIntent(recordValue.getIntent());
+      queueId.ifPresent(finishRecord::setQueueId);
+
+      commandWriter.appendFollowUpCommand(
+          distributionKey, CommandDistributionIntent.FINISH, finishRecord);
+    }
+  }
+
+  public void onFinish(
+      final long distributionKey, final CommandDistributionRecord distributionRecord) {
+
+    Optional.ofNullable(distributionRecord.getQueueId()).ifPresent(this::continueAfterQueue);
+
+    stateWriter.appendFollowUpEvent(
+        distributionKey, CommandDistributionIntent.FINISHED, distributionRecord);
+
+    getMetrics().removeActiveDistribution();
+  }
+
+  public void onScheduledRetry(
+      final long distributionKey, final CommandDistributionRecord distributionRecord) {
+
+    getMetrics().retryInflightDistribution(distributionRecord.getPartitionId());
+
+    getCommandSender()
+        .sendCommand(
+            distributionRecord.getPartitionId(),
+            distributionRecord.getValueType(),
+            distributionRecord.getIntent(),
+            distributionKey,
+            distributionRecord.getCommandValue());
+  }
+
+  public void foreachRetriableDistribution(final PendingDistributionVisitor consumer) {
+    distributionState.foreachRetriableDistribution(consumer);
+  }
+
   public interface RequestBuilder {
     DistributionRequestBuilder unordered();
 
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionFinishProcessor.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionFinishProcessor.java
index 346f1da15d81..d631c41aec68 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionFinishProcessor.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionFinishProcessor.java
@@ -9,31 +9,21 @@
 
 import io.camunda.zeebe.engine.processing.ExcludeAuthorizationCheck;
 import io.camunda.zeebe.engine.processing.streamprocessor.TypedRecordProcessor;
-import io.camunda.zeebe.engine.processing.streamprocessor.writers.StateWriter;
-import io.camunda.zeebe.engine.processing.streamprocessor.writers.Writers;
 import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
-import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
 import io.camunda.zeebe.stream.api.records.TypedRecord;
-import java.util.Optional;
 
 @ExcludeAuthorizationCheck
 public class CommandDistributionFinishProcessor
     implements TypedRecordProcessor<CommandDistributionRecord> {
   private final CommandDistributionBehavior commandDistributionBehavior;
-  private final StateWriter stateWriter;
 
   public CommandDistributionFinishProcessor(
-      final Writers writers, final CommandDistributionBehavior commandDistributionBehavior) {
-    stateWriter = writers.state();
+      final CommandDistributionBehavior commandDistributionBehavior) {
     this.commandDistributionBehavior = commandDistributionBehavior;
   }
 
   @Override
   public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
-    final var distributionRecord = record.getValue();
-    Optional.ofNullable(distributionRecord.getQueueId())
-        .ifPresent(commandDistributionBehavior::continueAfterQueue);
-    stateWriter.appendFollowUpEvent(
-        record.getKey(), CommandDistributionIntent.FINISHED, distributionRecord);
+    commandDistributionBehavior.onFinish(record.getKey(), record.getValue());
   }
 }
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
index bdc62ef2c08b..73dcdbe33ed8 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
@@ -7,9 +7,7 @@
  */
 package io.camunda.zeebe.engine.processing.distribution;
 
-import io.camunda.zeebe.engine.state.immutable.DistributionState;
 import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
-import io.camunda.zeebe.stream.api.InterPartitionCommandSender;
 import io.camunda.zeebe.stream.api.ReadonlyStreamProcessorContext;
 import io.camunda.zeebe.stream.api.StreamProcessorLifecycleAware;
 import java.time.Duration;
@@ -54,8 +52,7 @@ public final class CommandRedistributor implements StreamProcessorLifecycleAware
 
   private static final Logger LOG = LoggerFactory.getLogger(CommandRedistributor.class);
 
-  private final DistributionState distributionState;
-  private final InterPartitionCommandSender commandSender;
+  private final CommandDistributionBehavior distributionBehavior;
 
   /**
    * Tracks the number of attempted retry cycles for each retriable distribution. Note that this
@@ -64,10 +61,8 @@ public final class CommandRedistributor implements StreamProcessorLifecycleAware
    */
   private final Map<RetriableDistribution, Long> retryCyclesPerDistribution = new HashMap<>();
 
-  public CommandRedistributor(
-      final DistributionState distributionState, final InterPartitionCommandSender commandSender) {
-    this.distributionState = distributionState;
-    this.commandSender = commandSender;
+  public CommandRedistributor(final CommandDistributionBehavior distributionBehavior) {
+    this.distributionBehavior = distributionBehavior;
   }
 
   @Override
@@ -77,13 +72,18 @@ public void onRecovered(final ReadonlyStreamProcessorContext context) {
         .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
   }
 
-  private void runRetryCycle() {
+  public void runRetryCycle() {
     final var retriableDistributions = new HashSet<RetriableDistribution>();
-    distributionState.foreachRetriableDistribution(
+    distributionBehavior.foreachRetriableDistribution(
         (distributionKey, record) -> {
-          final var retriable = new RetriableDistribution(distributionKey, record.getPartitionId());
+          final var retriable = RetriableDistribution.from(distributionKey, record);
+          final Long retryCycle = updateRetryCycle(retriable);
+
+          if (retriable.shouldRetryNow(retryCycle)) {
+            retryDistribution(retriable, record, retryCycle);
+          }
+
           retriableDistributions.add(retriable);
-          retryDistribution(retriable, record);
           return true;
         });
 
@@ -91,51 +91,50 @@ private void runRetryCycle() {
     retryCyclesPerDistribution.keySet().removeIf(Predicate.not(retriableDistributions::contains));
   }
 
+  private Long updateRetryCycle(final RetriableDistribution retriable) {
+    return retryCyclesPerDistribution.compute(
+        retriable, (k, retryCycles) -> retryCycles != null ? retryCycles + 1 : 0L);
+  }
+
   private void retryDistribution(
       final RetriableDistribution retriable,
-      final CommandDistributionRecord commandDistributionRecord) {
-    if (!shouldRetryNow(retriable)) {
-      return;
-    }
+      final CommandDistributionRecord commandDistributionRecord,
+      final Long retryCycle) {
 
     LOG.info(
-        "Retrying to distribute retriable command {} ({}.{}) to partition {}",
+        "Retrying to distribute retriable command {} ({}.{}) to partition {} (Cycle: #{})",
         retriable.distributionKey,
         commandDistributionRecord.getValueType(),
         commandDistributionRecord.getIntent(),
-        retriable.partitionId);
-
-    commandSender.sendCommand(
         retriable.partitionId,
-        commandDistributionRecord.getValueType(),
-        commandDistributionRecord.getIntent(),
-        retriable.distributionKey,
-        commandDistributionRecord.getCommandValue());
+        retryCycle);
+
+    distributionBehavior.onScheduledRetry(retriable.distributionKey, commandDistributionRecord);
   }
 
-  /**
-   * Returns whether a retriable distribution should be retried now, or not in this cycle.
-   *
-   * <p>Calling this method increments the retry cycles tracking ({@link
-   * #retryCyclesPerDistribution}) of that retriable distribution, or initiates it at 0. The number
-   * of cycles is used to implement a simple exponential backoff.
-   */
-  private boolean shouldRetryNow(final RetriableDistribution retriableDistribution) {
-    // retryCycles starts off at 0, ensuring that we wait between COMMAND_REDISTRIBUTION_INTERVAL
-    // and 2 * COMMAND_REDISTRIBUTION_INTERVAL before retrying distribution.
-    final long retryCycle =
-        retryCyclesPerDistribution.compute(
-            retriableDistribution, (k, retryCycles) -> retryCycles != null ? retryCycles + 1 : 0L);
-
-    if (retryCycle >= MAX_RETRY_CYCLES) {
-      // Retry in intervals of RETRY_MAX_BACKOFF_DURATION
-      return retryCycle % MAX_RETRY_CYCLES == 0;
-    } else {
-      // Retry in intervals of COMMAND_REDISTRIBUTION_INTERVAL
-      // The interval is doubling until we reached RETRY_MAX_BACKOFF_DURATION
-      return Long.bitCount(retryCycle) == 1;
+  private record RetriableDistribution(long distributionKey, int partitionId) {
+    /**
+     * Returns whether a retriable distribution should be retried now, or not in the given cycle.
+     *
+     * <p>The number of cycles is used to implement a simple exponential backoff.
+     */
+    private boolean shouldRetryNow(final Long retryCycle) {
+      // retryCycles starts off at 0, ensuring that we wait between COMMAND_REDISTRIBUTION_INTERVAL
+      // and 2 * COMMAND_REDISTRIBUTION_INTERVAL before retrying distribution.
+
+      if (retryCycle >= MAX_RETRY_CYCLES) {
+        // Retry in intervals of RETRY_MAX_BACKOFF_DURATION
+        return retryCycle % MAX_RETRY_CYCLES == 0;
+      } else {
+        // Retry in intervals of COMMAND_REDISTRIBUTION_INTERVAL
+        // The interval is doubling until we reached RETRY_MAX_BACKOFF_DURATION
+        return Long.bitCount(retryCycle) == 1;
+      }
     }
-  }
 
-  private record RetriableDistribution(long distributionKey, int partitionId) {}
+    public static RetriableDistribution from(
+        final long distributionKey, final CommandDistributionRecord record) {
+      return new RetriableDistribution(distributionKey, record.getPartitionId());
+    }
+  }
 }
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
index f5206d1be300..233e5c227926 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
@@ -247,11 +247,18 @@ public CommandDistributionRecord getCommandDistributionRecord(
 
     return new CommandDistributionRecord()
         .setPartitionId(partition)
+        .setQueueId(persistedDistribution.getQueueId().orElse(null))
         .setValueType(persistedDistribution.getValueType())
         .setIntent(persistedDistribution.getIntent())
         .setCommandValue(persistedDistribution.getCommandValue());
   }
 
+  @Override
+  public void foreachCommandDistribution(final CommandDistributionVisitor visitor) {
+    commandDistributionRecordColumnFamily.whileTrue(
+        (compositeKey, record) -> visitor.visit(distributionKey.getValue(), record));
+  }
+
   @Override
   public void foreachRetriableDistribution(final PendingDistributionVisitor visitor) {
     final var lastDistributionKey = new MutableLong(0);
diff --git a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/immutable/DistributionState.java b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/immutable/DistributionState.java
index 2b1e407d6473..e9c49a758e14 100644
--- a/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/immutable/DistributionState.java
+++ b/zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/immutable/DistributionState.java
@@ -7,6 +7,7 @@
  */
 package io.camunda.zeebe.engine.state.immutable;
 
+import io.camunda.zeebe.engine.state.distribution.PersistedCommandDistribution;
 import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
 import java.util.Optional;
 
@@ -58,6 +59,13 @@ public interface DistributionState {
    */
   CommandDistributionRecord getCommandDistributionRecord(long distributionKey, int partition);
 
+  /**
+   * Visits each persisted command distribution {@link PersistedCommandDistribution}
+   *
+   * @param visitor Each distribution is visited by this visitor
+   */
+  void foreachCommandDistribution(CommandDistributionVisitor visitor);
+
   /**
    * Visits each persisted retriable distribution, providing both the key of that distribution and
    * the {@link CommandDistributionRecord}.
@@ -126,6 +134,13 @@ interface PendingDistributionVisitor {
     boolean visit(final long distributionKey, final CommandDistributionRecord pendingDistribution);
   }
 
+  /** This visitor can visit pending distributions of {@link CommandDistributionRecord}. */
+  @FunctionalInterface
+  interface CommandDistributionVisitor {
+    boolean visit(
+        final long distributionKey, final PersistedCommandDistribution pendingDistribution);
+  }
+
   @FunctionalInterface
   interface ContinuationCommandVisitor {
     /** Visits a registered continuation command. */
diff --git a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
new file mode 100644
index 000000000000..067341c56356
--- /dev/null
+++ b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
@@ -0,0 +1,242 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(0),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics ->
+                assertThat(metrics.count)
+                    .describedAs("Counters will not be recovered and are expected to reset to zero")
+                    .isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero());
+  }
+
+  private long triggerUnqueuedDistribution() {
+    // clock reset is used to trigger a command within the engine that requires distribution
+    // Note: this is not affecting the actual actor clock!
+    final var reset = engine.clock().reset();
+    waitUntilCommandDistributionIs(CommandDistributionIntent.DISTRIBUTING, reset.getKey());
+    return reset.getKey();
+  }
+
+  private long triggerQueuedDistribution() {
+    final var deploy =
+        engine
+            .deployment()
+            .withXmlResource(
+                "process.bpmn", Bpmn.createExecutableProcess().startEvent().endEvent().done())
+            .expectCreated()
+            .deploy();
+
+    waitUntilCommandDistributionIs(CommandDistributionIntent.ENQUEUED, deploy.getKey());
+
+    return deploy.getKey();
+  }
+
+  private static void waitUntilCommandDistributionIs(
+      final CommandDistributionIntent state, final long key) {
+    RecordingExporter.commandDistributionRecords(state).withRecordKey(key).await();
+  }
+
+  private void waitUntilDistributionIsRetried(final long key) {
+    RecordingExporter.setMaximumWaitTime(100);
+    Awaitility.await()
+        .untilAsserted(
+            () -> {
+              // wait for retry mechanism to trigger second distribution
+              // (while we intercepted the first acknowledgement)
+              engine.getClock().addTime(CommandRedistributor.COMMAND_REDISTRIBUTION_INTERVAL);
+
+              // Make sure we have two records on the target partition
+              assertThat(RecordingExporter.records().withPartitionId(2).withRecordKey(key).limit(2))
+                  .hasSizeGreaterThan(1);
+            });
+    RecordingExporter.setMaximumWaitTime(5000);
+  }
+
+  private MetricSnapshot snapshotMetrics() {
+    return new MetricSnapshot(
+        getCounterValue(1, "zeebe.command.distributions"),
+        getGaugeValue(1, "zeebe.command.distributions.active"),
+        getGaugeValue(1, "zeebe.command.distributions.pending"),
+        getGaugeValue(1, "zeebe.command.distributions.inflight"),
+        getCounterValue(1, "zeebe.command.distributions.inflight.retries"),
+        getCounterValue(1, "zeebe.command.distributions.acknowledged.received"),
+        getCounterValue(2, "zeebe.command.distributions.acknowledged.sent"));
+  }
+
+  private double getCounterValue(final int partition, final String metricName) {
+    final var counter = engine.getMeterRegistry(partition).find(metricName).counter();
+    return counter != null ? counter.count() : 0.0;
+  }
+
+  private double getGaugeValue(final int partition, final String metricName) {
+    final var gauge = engine.getMeterRegistry(partition).find(metricName).gauge();
+    return gauge != null ? gauge.value() : 0.0;
+  }
+
+  record MetricSnapshot(
+      double count,
+      double active,
+      double pending,
+      double inflight,
+      double retries,
+      double ackReceived,
+      double ackSent) {}
+}
diff --git a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehaviorTest.java b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehaviorTest.java
index 726ce1fd777a..798cbcbb0009 100644
--- a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehaviorTest.java
+++ b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehaviorTest.java
@@ -16,6 +16,7 @@
 import static org.mockito.Mockito.verifyNoMoreInteractions;
 import static org.mockito.Mockito.when;
 
+import io.camunda.zeebe.engine.metrics.DistributionMetrics;
 import io.camunda.zeebe.engine.processing.streamprocessor.writers.Writers;
 import io.camunda.zeebe.engine.state.appliers.EventAppliers;
 import io.camunda.zeebe.engine.state.immutable.DistributionState;
@@ -35,6 +36,7 @@
 import java.util.Set;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
 import org.junit.jupiter.api.Test;
 
 /**
@@ -53,6 +55,7 @@
 class CommandDistributionBehaviorTest {
 
   private DistributionState mockDistributionState;
+  private DistributionMetrics mockDistributionMetrics;
   private FakeProcessingResultBuilder<CommandDistributionRecord> fakeProcessingResultBuilder;
   private InterPartitionCommandSender mockInterpartitionCommandSender;
   private Writers writers;
@@ -64,6 +67,7 @@ class CommandDistributionBehaviorTest {
 
   @BeforeEach
   void setUp() {
+    mockDistributionMetrics = mock(DistributionMetrics.class);
     mockDistributionState = mock(DistributionState.class);
     fakeProcessingResultBuilder = new FakeProcessingResultBuilder<>();
     mockInterpartitionCommandSender = mock(InterPartitionCommandSender.class);
@@ -86,7 +90,8 @@ void shouldNotDistributeCommandToThisPartition() {
             writers,
             1,
             RoutingInfo.forStaticPartitions(1),
-            mockInterpartitionCommandSender);
+            mockInterpartitionCommandSender,
+            mockDistributionMetrics);
 
     // when distributing to all partitions
     behavior.withKey(key).unordered().distribute(command);
@@ -108,7 +113,8 @@ void shouldDistributeCommandToAllOtherPartitions() {
             writers,
             1,
             RoutingInfo.forStaticPartitions(3),
-            mockInterpartitionCommandSender);
+            mockInterpartitionCommandSender,
+            mockDistributionMetrics);
 
     // when distributing to all partitions
     behavior.withKey(key).unordered().distribute(command);
@@ -144,7 +150,8 @@ void shouldDistributeCommandToSpecificPartitions() {
             writers,
             2,
             RoutingInfo.forStaticPartitions(4),
-            mockInterpartitionCommandSender);
+            mockInterpartitionCommandSender,
+            mockDistributionMetrics);
 
     // when distributing to partitions 1 and 3
     behavior.withKey(key).unordered().forPartitions(Set.of(1, 3)).distribute(command);
@@ -180,7 +187,8 @@ void shouldStartQueueImmediately() {
             writers,
             1,
             RoutingInfo.forStaticPartitions(3),
-            mockInterpartitionCommandSender);
+            mockInterpartitionCommandSender,
+            mockDistributionMetrics);
 
     // when distributing first command in queue to all partitions
     behavior.withKey(key).inQueue("test-queue").distribute(command);
@@ -214,7 +222,8 @@ void shouldWaitInQueue() {
             writers,
             1,
             RoutingInfo.forStaticPartitions(3),
-            mockInterpartitionCommandSender);
+            mockInterpartitionCommandSender,
+            mockDistributionMetrics);
 
     final var firstKey = Protocol.encodePartitionId(1, 100);
     final var secondKey = Protocol.encodePartitionId(1, 101);
@@ -248,4 +257,71 @@ void shouldWaitInQueue() {
         .sendCommand(eq(3), eq(valueType), eq(intent), eq(firstKey), any());
     verifyNoMoreInteractions(mockInterpartitionCommandSender);
   }
+
+  @Nested
+  class OnAcknowledge {
+    CommandDistributionBehavior behavior;
+
+    @BeforeEach
+    void setUp() {
+      behavior =
+          new CommandDistributionBehavior(
+              mockDistributionState,
+              writers,
+              1,
+              RoutingInfo.forStaticPartitions(2),
+              mockInterpartitionCommandSender,
+              mockDistributionMetrics);
+    }
+
+    @Test
+    public void shouldAcknowledgeCommandAndFinishDistribution() {
+      final CommandDistributionRecord record = new CommandDistributionRecord().setPartitionId(2);
+
+      when(mockDistributionState.hasPendingDistribution(123L)).thenReturn(false);
+
+      behavior.onAcknowledgeDistribution(123L, record);
+
+      Assertions.assertThat(fakeProcessingResultBuilder.getFollowupRecords())
+          .extracting(Record::getKey, Record::getIntent, r -> r.getValue().getPartitionId())
+          .containsExactly(
+              tuple(123L, CommandDistributionIntent.ACKNOWLEDGED, 2),
+              tuple(123L, CommandDistributionIntent.FINISH, 1));
+    }
+
+    @Test
+    public void shouldAcknowledgeCommandAndNotFinishDistribution() {
+      final CommandDistributionRecord record = new CommandDistributionRecord().setPartitionId(2);
+
+      when(mockDistributionState.hasPendingDistribution(123L)).thenReturn(true);
+
+      behavior.onAcknowledgeDistribution(123L, record);
+
+      Assertions.assertThat(fakeProcessingResultBuilder.getFollowupRecords())
+          .extracting(Record::getKey, Record::getIntent, r -> r.getValue().getPartitionId())
+          .containsExactly(tuple(123L, CommandDistributionIntent.ACKNOWLEDGED, 2));
+    }
+
+    @Test
+    public void shouldAcknowledgeCommandAndContinueQueue() {
+      final CommandDistributionRecord record = new CommandDistributionRecord().setPartitionId(2);
+      final CommandDistributionRecord otherRecord =
+          new CommandDistributionRecord().setPartitionId(2);
+
+      when(mockDistributionState.getQueueIdForDistribution(123L)).thenReturn(Optional.of("queue"));
+      when(mockDistributionState.getNextQueuedDistributionKey("queue", 2))
+          .thenReturn(Optional.of(124L));
+      when(mockDistributionState.getCommandDistributionRecord(124L, 2)).thenReturn(otherRecord);
+      when(mockDistributionState.hasPendingDistribution(123L)).thenReturn(false);
+
+      behavior.onAcknowledgeDistribution(123L, record);
+
+      Assertions.assertThat(fakeProcessingResultBuilder.getFollowupRecords())
+          .extracting(Record::getKey, Record::getIntent, r -> r.getValue().getPartitionId())
+          .containsExactly(
+              tuple(123L, CommandDistributionIntent.ACKNOWLEDGED, 2),
+              tuple(124L, CommandDistributionIntent.DISTRIBUTING, 2),
+              tuple(123L, CommandDistributionIntent.FINISH, 1));
+    }
+  }
 }
diff --git a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributorTest.java b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributorTest.java
new file mode 100644
index 000000000000..9ea99cff4cb8
--- /dev/null
+++ b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributorTest.java
@@ -0,0 +1,71 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.processing.distribution;
+
+import static org.junit.jupiter.api.Assertions.*;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+
+import io.camunda.zeebe.engine.state.immutable.DistributionState.PendingDistributionVisitor;
+import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
+import org.junit.Before;
+import org.junit.Test;
+
+public class CommandRedistributorTest {
+
+  private CommandDistributionBehavior behavior;
+
+  @Before
+  public void setUp() {
+    behavior = mock(CommandDistributionBehavior.class);
+  }
+
+  @Test
+  public void shouldNotRetryOnFirstCycle() {
+    final var redistributor = new CommandRedistributor(behavior);
+    final var distributionRecord = new CommandDistributionRecord().setPartitionId(2);
+
+    doAnswer(
+            invocation -> {
+              return invocation
+                  .<PendingDistributionVisitor>getArgument(0)
+                  .visit(123L, distributionRecord);
+            })
+        .when(behavior)
+        .foreachRetriableDistribution(any(PendingDistributionVisitor.class));
+
+    redistributor.runRetryCycle();
+
+    verify(behavior, never()).onScheduledRetry(123L, distributionRecord);
+  }
+
+  @Test
+  public void shouldRetryOnSecondCycle() {
+    final var redistributor = new CommandRedistributor(behavior);
+    final var distributionRecord = new CommandDistributionRecord().setPartitionId(2);
+
+    doAnswer(
+            invocation -> {
+              return invocation
+                  .<PendingDistributionVisitor>getArgument(0)
+                  .visit(123L, distributionRecord);
+            })
+        .when(behavior)
+        .foreachRetriableDistribution(any(PendingDistributionVisitor.class));
+
+    redistributor.runRetryCycle();
+    redistributor.runRetryCycle();
+
+    verify(behavior).onScheduledRetry(123L, distributionRecord);
+  }
+
+  private void addDistribution(final CommandDistributionRecord... records) {}
+}
diff --git a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/message/MessageStreamProcessorTest.java b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/message/MessageStreamProcessorTest.java
index 5786dc8d1a4b..b6dc44b6a9ed 100644
--- a/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/message/MessageStreamProcessorTest.java
+++ b/zeebe/engine/src/test/java/io/camunda/zeebe/engine/processing/message/MessageStreamProcessorTest.java
@@ -21,6 +21,7 @@
 import static org.mockito.Mockito.when;
 
 import io.camunda.zeebe.engine.EngineConfiguration;
+import io.camunda.zeebe.engine.metrics.DistributionMetrics;
 import io.camunda.zeebe.engine.processing.bpmn.behavior.BpmnBehaviors;
 import io.camunda.zeebe.engine.processing.distribution.CommandDistributionBehavior;
 import io.camunda.zeebe.engine.processing.identity.AuthorizationCheckBehavior;
@@ -84,7 +85,8 @@ public void setup() {
                 writers,
                 1,
                 RoutingInfo.forStaticPartitions(1),
-                mockInterpartitionCommandSender));
+                mockInterpartitionCommandSender,
+                mock(DistributionMetrics.class)));
 
     rule.startTypedStreamProcessor(
         (typedRecordProcessors, processingContext) -> {


Reviews:
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T11:08:59Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T11:09:59Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T11:12:00Z
user: login: remcowesterhoud
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T08:15:37Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:07:00Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:08:49Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:09:55Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:15:34Z
user: login: remcowesterhoud
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:25:54Z
user: login: remcowesterhoud
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T07:32:35Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T08:03:55Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T08:05:30Z
user: login: remcowesterhoud
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T08:13:39Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T09:43:45Z
user: login: npepinpe
body: :+1: I like the approach of using a dummy interface for discovery, hope to see more of this spread out :)

:thought_balloon: As you know I'm not a fan of having metrics in the state, especially gauges, since we sometimes have multiple instances of the state classes, so what is the "true" value? I would avoid putting this in the state, and instead add it to the behavior if possible. The `onRecovered` of the state can replicated by the general engine's `onRecovered`, for example (where you can delegate to other components).

:thought_balloon: That said, I understand it's very useful, so I'm not saying it's forbidden or anything :smile: 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T11:10:14Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T12:19:08Z
user: login: npepinpe
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T12:20:02Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T12:33:36Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T12:36:58Z
user: login: remcowesterhoud
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T07:45:31Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:05:10Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:06:23Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T15:14:09Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T10:03:59Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T10:06:34Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T09:16:50Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T09:18:37Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T09:20:04Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T09:27:01Z
user: login: remcowesterhoud
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T08:41:33Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T09:50:31Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T09:51:39Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T09:53:03Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T10:04:50Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T10:08:50Z
user: login: entangled90
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:22:17Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:29:22Z
user: login: entangled90
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:32:29Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:36:15Z
user: login: entangled90
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:50:24Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:54:49Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T08:57:31Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:04:55Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:16:56Z
user: login: entangled90
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:28:06Z
user: login: entangled90
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:32:28Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T10:28:43Z
user: login: korthout
body: Great work @eppdot 🏗️ 

👍 I really like the direction you've taken this. It makes parts of the feature easier to understand.

:x: I have a few comments and questions. Please have a look.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-21T14:26:43Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:28:53Z
user: login: korthout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T15:55:16Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T06:08:32Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T06:11:08Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T06:13:16Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T06:15:41Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T06:16:45Z
user: login: korthout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T09:36:29Z
user: login: korthout
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T09:50:13Z
user: login: korthout
body: Thanks for the changes @eppdot 🚀 

🔧 I'm not convinced that the reuse of the behavior class between the two actors is a good idea. We see this in the need for getters to expose the internals of the behavior class, and a `.withState` method that creates a new instance of the class but is only useful for constructing an instance of it. I worry a bit that the reuse leads to concurrency issues between the two actors, although they seem to be averted for now. I suggest to split the logic more strictly.

LGTM 👍 

❓ Will you also add new Grafana panels?


state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-22T12:21:28Z
user: login: eppdot
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T05:49:25Z

Review comments:
diff_hunk: @@ -52,6 +52,8 @@ public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
     final var partitionId = recordValue.getPartitionId();
 
     if (!distributionState.hasPendingDistribution(distributionKey, partitionId)) {
+      distributionState.getMetrics().noPendingDistribution(partitionId, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
user: login: eppdot
body: this metric has to be proven useful, I have not decided yet 😅 
created_at: 2025-03-19T11:08:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: this needs to be shaped out and it might be very performance heavy to iterate over all partitions + distributions + column families
created_at: 2025-03-19T11:09:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,40 +134,54 @@ public void addCommandDistribution(
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.insert(
         this.distributionKey, new PersistedCommandDistribution().wrap(commandDistributionRecord));
+
+    getMetrics().addDistribution(distributionKey);
   }
 
   @Override
   public void removeCommandDistribution(final long distributionKey) {
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.deleteIfExists(this.distributionKey);
+
+    getMetrics().removeDistribution(distributionKey);
   }
 
   @Override
   public void addRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.insert(distributionPartitionKey, DbNil.INSTANCE);
+
+    getMetrics().addInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void removeRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.deleteExisting(distributionPartitionKey);
+
+    getMetrics().removeInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void addPendingDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     pendingDistributionColumnFamily.upsert(distributionPartitionKey, DbNil.INSTANCE);
+
+    // TODO: the upsert above makes counting metrics difficult as this is currently called twice
+    // (ENQUEUED/DISTRIBUTING) and therefore would also be counted twice.
+    getMetrics().addPendingDistribution(partition, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: here we need to find a solution to prevent counting the `add` twice, e.g. by clearly distinguishing between queue vs no-queue strategy.
created_at: 2025-03-19T11:12:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: remcowesterhoud
body: I see your concerns for adding it. Personally I'd say let's not add it as it should be possible to derive this data from the other metrics. If we feel the need we can always add it later.
created_at: 2025-03-20T07:49:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: remcowesterhoud
body: ❓ I'm not overly familiar with the metrics so bear with me if I say something stupid. It reads to me like we will time the redistributions. E.g. how long does it take for this redistribution to be completed. What I'd be interested in is slightly different. I'd like to know how long it takes from the creation of the distribution (pending or immediately in-flight) to the acknowledgement of the distribution (so per partition).
created_at: 2025-03-20T08:03:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: You could simplify this by using `pendingDistributionColumnFamily.count()`. However as listed on the javadoc of that method, and as you already identified.

> Count the number of entries in the column family by iterating over all its entries. This is an
expensive operation and should be used with care.
created_at: 2025-03-20T08:06:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,40 +134,54 @@ public void addCommandDistribution(
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.insert(
         this.distributionKey, new PersistedCommandDistribution().wrap(commandDistributionRecord));
+
+    getMetrics().addDistribution(distributionKey);
   }
 
   @Override
   public void removeCommandDistribution(final long distributionKey) {
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.deleteIfExists(this.distributionKey);
+
+    getMetrics().removeDistribution(distributionKey);
   }
 
   @Override
   public void addRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.insert(distributionPartitionKey, DbNil.INSTANCE);
+
+    getMetrics().addInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void removeRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.deleteExisting(distributionPartitionKey);
+
+    getMetrics().removeInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void addPendingDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     pendingDistributionColumnFamily.upsert(distributionPartitionKey, DbNil.INSTANCE);
+
+    // TODO: the upsert above makes counting metrics difficult as this is currently called twice
+    // (ENQUEUED/DISTRIBUTING) and therefore would also be counted twice.
+    getMetrics().addPendingDistribution(partition, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: We could look into removing the duplicate `addPendingDistribution` call. I can't recall why have this.
created_at: 2025-03-20T08:15:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: Yeah, I am unsure if this is feasible, as we need to do `families x partitions (x queues?)` if we want to have counts per queue 🤔 
created_at: 2025-03-21T07:07:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -117,40 +134,54 @@ public void addCommandDistribution(
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.insert(
         this.distributionKey, new PersistedCommandDistribution().wrap(commandDistributionRecord));
+
+    getMetrics().addDistribution(distributionKey);
   }
 
   @Override
   public void removeCommandDistribution(final long distributionKey) {
     this.distributionKey.wrapLong(distributionKey);
     commandDistributionRecordColumnFamily.deleteIfExists(this.distributionKey);
+
+    getMetrics().removeDistribution(distributionKey);
   }
 
   @Override
   public void addRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.insert(distributionPartitionKey, DbNil.INSTANCE);
+
+    getMetrics().addInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void removeRetriableDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     retriableDistributionColumnFamily.deleteExisting(distributionPartitionKey);
+
+    getMetrics().removeInflightDistribution(partition, distributionKey);
   }
 
   @Override
   public void addPendingDistribution(final long distributionKey, final int partition) {
     this.distributionKey.wrapLong(distributionKey);
     partitionKey.wrapInt(partition);
     pendingDistributionColumnFamily.upsert(distributionPartitionKey, DbNil.INSTANCE);
+
+    // TODO: the upsert above makes counting metrics difficult as this is currently called twice
+    // (ENQUEUED/DISTRIBUTING) and therefore would also be counted twice.
+    getMetrics().addPendingDistribution(partition, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: Yup, I wanted to think about how to make this more robust, thus it's harder to break this again by accident. e.g. is there a way to know from the `upsert` if it was an insert or delete 🤔 
created_at: 2025-03-21T07:08:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: Was also my feeling mentioned [here](https://github.com/camunda/camunda/pull/29804#discussion_r2003079042)
created_at: 2025-03-21T07:09:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: eppdot
body: Yes, this one was especially to monitor how long the whole redestribution mechanism needs. As we loop over all entries and if this builds up, this might get unbearable slow and block other tasks from running in this actor/thread. Thus I thought about helping us detect this situations, e.g. via a [Long Task Timer](https://docs.micrometer.io/micrometer/reference/concepts/long-task-timers.html)

--- 

I have the same interest as you 😄 to measure request latency and it's on my list 🏃 
I currently see two ways:
1. Use [Timer Samples](https://docs.micrometer.io/micrometer/reference/concepts/timers.html#_storing_start_state_in_timer_sample) which allows for flexible measurement, but requires the metric registry to use more in memory storage to hold the samples (should be fine in general as we have number of queues + all unordered distributions inflight)
2. or add a `created_at` entry to all command distribution entries, thus we can calculate roundtrip time based on this and can also do other things (like visualizing age of entries etc.)

_Sidenote_: I was also thinking if all distributions should be done via queue, that would bound the number of overall inflight distributions.
created_at: 2025-03-21T07:15:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: Do you know how we do this for other metrics? 
created_at: 2025-03-21T07:25:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: remcowesterhoud
body: > I was also thinking if all distributions should be done via queue, that would bound the number of overall inflight distributions.

I'm not against this as it streamlines the whole distribution cycle. We currently have 4 places where we are not using a queue:
1. BatchOperationCreateProcessor
2. ClockProcessor
3. ProcessInstanceMigrationCatchEventBehaviour
4. SignalBroadcastProcessor

The advantage of not having a queue is that there is no waiting for other distributions. The same could be achieved by ensuring a unique queue. At the non-queued distributions we likely always have a key available which we could consider using for the queue.
created_at: 2025-03-21T07:32:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: Yes, for example `BannedInstanceMetrics`, but its only checking one column family

```  @Override
  public void onRecovered(final ReadonlyStreamProcessorContext context) {
    final var counter = new AtomicInteger(0);
    bannedInstanceColumnFamily.forEach(ignore -> counter.getAndIncrement());
    bannedInstanceMetrics.setBannedInstanceCounter(counter.get());
  }
```
created_at: 2025-03-21T08:03:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: eppdot
body: Thus you would "create" a queue for each single distribution or would you create 4 queues (`batch`, `clock`, `migration`, `signal`)?
created_at: 2025-03-21T08:05:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: remcowesterhoud
body: I would create a queue for each single distribution, because that will keep the current behaviour the same. Eg `distributionBehavior.withKey(key).inQueue(String.valueOf(key)).distribute(command);`

And to keep it convenient and abstract this away we can just modify the `unordered()` method:

```java
    @Override
    public DistributionRequest unordered() {
      queue = String.valueOf(key);
      return this;
    }
```
created_at: 2025-03-21T08:13:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: @npepinpe do you think it's fine to iterate like this? If we wanna use gauges, I don't have another idea how to do it nicely, apart from storing the statistics/count in another column family. 

One further alternative would be to use two counters (one for the `add` operation and one for the `remove` operation per column family) which might mimic a gauge (server side, we can kinda subtract these values from each other). But counters are rather used to describe the rate of change instead of real counts (e.g. they could be reset at any point). Thus, what we could use them for, would be to check if more elements are added to the queue than removed 🤔 

(_disclaimer: I am just learning about these details, thus maybe I am wrong_)
created_at: 2025-03-21T09:43:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: npepinpe
body: I think it's OK to do an expensive operation on recovered - this is not the hot path, since it's only called very seldom.
created_at: 2025-03-21T10:52:47Z
author_association: MEMBER
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: npepinpe
body: As you mentioned, another alternative is storing aggregations in the state, and synchronizing them. That's a valid alternative, but I don't think we have any evidence this is something we need right now. 

Do we expect this count would ever be very high (e.g. hundreds of thousands)?
created_at: 2025-03-21T10:55:42Z
author_association: MEMBER
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: npepinpe
body: About the latency measurement, up until now, we typically would measure via different metrics (e.g. measure latency of distribution creation, then distribution per partition), and then sum them up in Grafana. The `LongTaskTimer` is interesting, but we haven't used it before, so we should try it and see if it works for us - how does it map to Prometheus, how do we use it in Grafana, etc.
created_at: 2025-03-21T10:59:05Z
author_association: MEMBER
diff_hunk: @@ -52,6 +52,8 @@ public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
     final var partitionId = recordValue.getPartitionId();
 
     if (!distributionState.hasPendingDistribution(distributionKey, partitionId)) {
+      distributionState.getMetrics().noPendingDistribution(partitionId, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
user: login: npepinpe
body: What would it be for?
created_at: 2025-03-21T10:59:18Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: npepinpe
body: :question: Why always pass the keys? We wouldn't actually add this as a label/tag, since the cardinality is very very high, right?
created_at: 2025-03-21T11:09:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
+
+  public void addInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addInflightDistribution(distributionKey);
+  }
+
+  public void removeInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removeInflightDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void acknowledgeDistribution(final int partitionId, final long distributionKey) {}
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See CommandRedistribution.java
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void retryInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).retryInflightDistribution(distributionKey);
+  }
+
+  public PartitionDistributionMetrics getPartitionMetrics(final int partitionId) {
+    return partitionMetrics.computeIfAbsent(
+        partitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  public void timedRedistribution(final Runnable redistribution) {
+    // timer
+    redistribution.run();
+  }
+
+  private class PartitionDistributionMetrics {
+
+    private final MeterRegistry meterRegistry;
+    private final int partitionId;
+
+    public PartitionDistributionMetrics(final int partitionId, final MeterRegistry meterRegistry) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: npepinpe
body: :upside_down_face: The meter registry given here already has a common partition ID tag, so you can assume _all_ your metrics will have a partition tag added, so you don't need to track it.
created_at: 2025-03-21T11:10:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
+
+  public void addInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addInflightDistribution(distributionKey);
+  }
+
+  public void removeInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removeInflightDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void acknowledgeDistribution(final int partitionId, final long distributionKey) {}
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See CommandRedistribution.java
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void retryInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).retryInflightDistribution(distributionKey);
+  }
+
+  public PartitionDistributionMetrics getPartitionMetrics(final int partitionId) {
+    return partitionMetrics.computeIfAbsent(
+        partitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  public void timedRedistribution(final Runnable redistribution) {
+    // timer
+    redistribution.run();
+  }
+
+  private class PartitionDistributionMetrics {
+
+    private final MeterRegistry meterRegistry;
+    private final int partitionId;
+
+    public PartitionDistributionMetrics(final int partitionId, final MeterRegistry meterRegistry) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: Not exactly, this is the partition id of the target partition 😄 the one in the MeterRegistry is the id of the source/origin partition. This way we see how many outgoing distributions per target partition we have, e.g. when partition X is piling up, but others are processing distributions well.

Sure, it's up for discussion, if we need this granularity 🤷 
created_at: 2025-03-21T12:19:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
+
+  public void addInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addInflightDistribution(distributionKey);
+  }
+
+  public void removeInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removeInflightDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void acknowledgeDistribution(final int partitionId, final long distributionKey) {}
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See CommandRedistribution.java
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void retryInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).retryInflightDistribution(distributionKey);
+  }
+
+  public PartitionDistributionMetrics getPartitionMetrics(final int partitionId) {
+    return partitionMetrics.computeIfAbsent(
+        partitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  public void timedRedistribution(final Runnable redistribution) {
+    // timer
+    redistribution.run();
+  }
+
+  private class PartitionDistributionMetrics {
+
+    private final MeterRegistry meterRegistry;
+    private final int partitionId;
+
+    public PartitionDistributionMetrics(final int partitionId, final MeterRegistry meterRegistry) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: npepinpe
body: Ah, right, that wasn't clear to me but it makes perfect sense :)
created_at: 2025-03-21T12:20:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
+    // gauge  + counter?
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    // gauge
+  }
+
+  public void addPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removePendingDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a pending distribution is not found for a received acknowledgement.
+   * This may happen if a distribution has been retried, but in the meantime an acknowledgement was
+   * already received - thus this was duplicate acknowledgement.
+   *
+   * <p>TBD if this metrics brings additional value, compared to the retryInflightDistribution
+   * metric as it also represents any delay/issues between origin and target partition.
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void noPendingDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).noPendingDistribution(distributionKey);
+  }
+
+  public void addInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).addInflightDistribution(distributionKey);
+  }
+
+  public void removeInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).removeInflightDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void acknowledgeDistribution(final int partitionId, final long distributionKey) {}
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See CommandRedistribution.java
+   *
+   * @param partitionId the partition id of the distribution
+   * @param distributionKey the key of the distribution
+   */
+  public void retryInflightDistribution(final int partitionId, final long distributionKey) {
+    getPartitionMetrics(partitionId).retryInflightDistribution(distributionKey);
+  }
+
+  public PartitionDistributionMetrics getPartitionMetrics(final int partitionId) {
+    return partitionMetrics.computeIfAbsent(
+        partitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  public void timedRedistribution(final Runnable redistribution) {
+    // timer
+    redistribution.run();
+  }
+
+  private class PartitionDistributionMetrics {
+
+    private final MeterRegistry meterRegistry;
+    private final int partitionId;
+
+    public PartitionDistributionMetrics(final int partitionId, final MeterRegistry meterRegistry) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: naming is hard, maybe we better call it `targetPartitionId` or `receiverPartitionId` 🙃 
created_at: 2025-03-21T12:33:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: I am not sure if it is needed - that will clear up when implementing the actual metrics. I can imagine it would help correlating `add` and `remove` events, e.g. for timers on the client side or server side to do latency calculations there (I remember from datadog, that you can also send raw events and correlate them based on some payload, like the distributionKey)
created_at: 2025-03-21T12:36:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: > Do we expect this count would ever be very high (e.g. hundreds of thousands)?

No, in a healthy cluster this shouldn't become very high. In theory it'll only grow big if the other partition(s) have a large processing queue, or if communication between brokers somehow fails for a longer period of time. Both of these things are exceptional cases with a large impact on the cluster. I'd question of the slightly longer recovery matters in those situations.
created_at: 2025-03-24T07:45:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -52,6 +52,8 @@ public void processRecord(final TypedRecord<CommandDistributionRecord> record) {
     final var partitionId = recordValue.getPartitionId();
 
     if (!distributionState.hasPendingDistribution(distributionKey, partitionId)) {
+      distributionState.getMetrics().noPendingDistribution(partitionId, distributionKey);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionAcknowledgeProcessor.java
user: login: eppdot
body: It describes the number of "acknowledgements" we get after we already concluded the distribution. This may happen when retries/redistributions are occurring, but I think I would leave this metric out for now - as we could get similar information out of the "number for retries" metric.
created_at: 2025-03-24T15:05:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +114,18 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    final var counter = new AtomicInteger(0);
+
+    // TODO: do it by partition and for each column families
+    // what about performance here? is it okay to take longer during recovery phase?
+    pendingDistributionColumnFamily.forEach(ignore -> counter.getAndIncrement());
+    getMetrics().reset(counter.get());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: Ok, I will move forward with doing the simple `count` per partition per columnfamily and if it causes issues in the future, we need to come up with a better solution.
created_at: 2025-03-24T15:06:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: eppdot
body: @remcowesterhoud I was also thinking to have metrics per queue, if we now create a temporary queue for each `unordered` distribution, this would make the metrics explode 😅 What we could do though is define a prefix for these kind of distribution and ignore the tmp queue name for metrics. e.g.

```java
  @Override
    public DistributionRequest unordered() {
      queue = "tmp:" + String.valueOf(key);
      return this;
    }
```

@npepinpe I think LongTaskTimer is not great for measuring the round trip time, but rather a long running task/loop. For the latency I would wether use [timer samples](https://docs.micrometer.io/micrometer/reference/concepts/timers.html#_storing_start_state_in_timer_sample) or store some dates in the distribution records (something like `created_at`, `started_distribution_at`, `last_retry_at` or so - the less the better though). I am leaning a bit towards the latter approach as it's more flexible, durable and paints a clear picture - we could also use this information elsewhere (e.g. retry calculations).
created_at: 2025-03-24T15:14:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -74,7 +74,12 @@ public CommandRedistributor(
   public void onRecovered(final ReadonlyStreamProcessorContext context) {
     context
         .getScheduleService()
-        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::runRetryCycle);
+        .runAtFixedRate(COMMAND_REDISTRIBUTION_INTERVAL, this::redistribute);
+  }
+
+  // TODO: maybe push this one level higher in the SchedulerService to monitor all tasks
+  public void redistribute() {
+    distributionState.getMetrics().timedRedistribution(this::runRetryCycle);
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: eppdot
body: I will follow up with both things in separate PRs
- measuring latency from distribution_created to `distribution_acknowleded`
- every distribution should have a (tmp) queue, which eases complexity (need for edge case)
created_at: 2025-04-29T10:03:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: We possibly need this or something similar for the timer/latency metrics (If not I will remove them with that PR)
created_at: 2025-04-29T10:06:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,6 +221,7 @@ private void startDistributing(
         commandDistributionDistributing
             .setPartitionId(partition)
             .setValueType(valueType)
+            .setQueueId(queueId)
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: eppdot
body: this was really hard to find - I figured some fields of the record are filled only sometimes
created_at: 2025-04-30T09:16:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: this one was quite tricky and I am not 100% certain if it's great. 🤯 

also the counters are all incremented during replay (they might also already been incremented before), thus we might count the events twice 🙃 (cc @npepinpe)
created_at: 2025-04-30T09:18:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -247,6 +289,7 @@ public CommandDistributionRecord getCommandDistributionRecord(
 
     return new CommandDistributionRecord()
         .setPartitionId(partition)
+        .setQueueId(persistedDistribution.getQueueId().orElse(""))
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: this one also took me some time to find 🕵️  I was wondering in my condition (to not count pending twice) why the heck the queueId was all of a sudden empty 😅 
created_at: 2025-04-30T09:20:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,256 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import io.micrometer.core.instrument.MeterRegistry;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: eppdot
body: this one is kinda tricky and I haven't found a great solution. Goal is to ensure that the state of the gauges are recovered and synced with what is in rocksdb (ensure consistency).

Ideally, I want to trigger recovery and then make an assertion after recovery is done, but before processing further (e.g. I have paused processing for partition 2, but after calling `start` it continues processing). Which is not possible, as the continued processing change the metrics 🙈 . Thus I can only assert that after recovery the gauges must have been at 1 and then through processing they have been decremented back to 0 (otherwise it would be -1).

Another observation is that before the recovery is finished, replay happens, which causes the counter to be incremented to 1. This might result in counting this distribution twice 🤔, but I assume we have this problem with all counter? cc @npepinpe  
created_at: 2025-04-30T09:27:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,152 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Ideas:
+ *
+ * <p>- TimeGauge to track the age of the current/oldest distribution? or last latency to another
+ * partition? Probably better a histogram /timer
+ *
+ * <p>TimeGauge is probably better for onRecovery Tracking???
+ *
+ * <p>- ... - ... - ...
+ */
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>(); // TODO check if concurrent map is needed
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    // TODO: setup micrometer metrics once (performance)
+  }
+
+  /**
+   * Resets the metrics (e.g. in case of recovery to remain in sync with the state). Be careful
+   * about race conditions and call this only from the stream processing actor
+   */
+  public void reset(final int counter) {}
+
+  public void addDistribution(final long distributionKey) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: remcowesterhoud
body: Since it is not used now, could we remove it and add when we need it in that PR instead of the other way around 🙂 
created_at: 2025-05-02T07:29:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,216 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution(final long distributionKey) {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution(final long distributionKey) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: remcowesterhoud
body: 🔧 Same here. Could we remove it and add it when we need it instead of the other way around.
created_at: 2025-05-02T07:31:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,216 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution(final long distributionKey) {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution(final long distributionKey) {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId, final long distributionKey) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution(distributionKey);
+  }
+
+  public void removePendingDistribution(final int targetPartitionId, final long distributionKey) {
+    getPartitionMetrics(targetPartitionId).removePendingDistribution(distributionKey);
+  }
+
+  public void addInflightDistribution(final int targetPartitionId, final long distributionKey) {
+    getPartitionMetrics(targetPartitionId).addInflightDistribution(distributionKey);
+  }
+
+  public void removeInflightDistribution(final int targetPartitionId, final long distributionKey) {
+    getPartitionMetrics(targetPartitionId).removeInflightDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed and an acknowledgement is sent back to the origin partition.
+   *
+   * @param originPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   * @param distributionKey the key of the distribution
+   */
+  public void sentAcknowledgeDistribution(final int originPartitionId, final long distributionKey) {
+    getPartitionMetrics(originPartitionId).sentAcknowledgeDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called on the origin partition when a distribution command has been successfully
+   * acknowledged by the target partition.
+   *
+   * @param targetPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   * @param distributionKey the key of the distribution
+   */
+  public void receivedAcknowledgeDistribution(
+      final int targetPartitionId, final long distributionKey) {
+    getPartitionMetrics(targetPartitionId).receivedAcknowledgeDistribution(distributionKey);
+  }
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See CommandRedistribution.java
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: remcowesterhoud
body: ```suggestion
   * See {@link CommandRedistribution}.
```

🔧 Let's link it properly. Otherwise if the class is renamed or whatever the doc becomes outdated.
created_at: 2025-05-02T07:34:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,6 +221,7 @@ private void startDistributing(
         commandDistributionDistributing
             .setPartitionId(partition)
             .setValueType(valueType)
+            .setQueueId(queueId)
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: remcowesterhoud
body: In general we quite often have empty properties in our records. For this one it doesn't impact much as it's small. But if you imagine something like a large variable we save space in our record batch size and run less risk of exceeding it.
created_at: 2025-05-02T07:42:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,7 +24,11 @@ public CommandDistributionDistributingApplier(final MutableDistributionState dis
 
   @Override
   public void applyState(final long key, final CommandDistributionRecord value) {
-    distributionState.addPendingDistribution(key, value.getPartitionId());
+    if (Strings.isEmpty(value.getQueueId())) {
+      // enqueued items are already added to the pending distributions
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingApplier.java
user: login: remcowesterhoud
body: ❌  This is a bit of a scary change. Applied events must be deterministic. By changing the applier it results in a different behavior when applying the event. Think of this situation:

1. Broker 1 applies event `X` without queue id before this change. A pending distribution is added to the state.
2. An update happens. Broker 2 is running the new version.
3. Event `X` is replicated to Broker 2. This uses the new event applier, as a result the pending distribution is not added to the state.
4. We have an incosistency.


Looking at the rest of the code I think it'll work fine as the pending distribution gets added twice. However, I still believe we'd better be safe than sorry and introduce a new version of this event applier.


created_at: 2025-05-02T08:18:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: ❓ Doesn't this method reset the metrics and rebuild it? So any events that happened during replay would be reset and rebuild from the ground up? How would they get counted twice?
created_at: 2025-05-02T08:20:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -247,6 +289,7 @@ public CommandDistributionRecord getCommandDistributionRecord(
 
     return new CommandDistributionRecord()
         .setPartitionId(partition)
+        .setQueueId(persistedDistribution.getQueueId().orElse(""))
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: remcowesterhoud
body: 🔧  I'd prefer changing this to `orElse(null)`. The `setQueueId` method in the `CommandDistributionRecord` has some logic when it's `null` or not. If it's `null` it resets the property, else it will set the value of the property. They both result in an empty string being set as the `queueId`. But if someone comes across this code in the future and wants to do something else if this is `null`, passing it along to the setter is more descriptive.
created_at: 2025-05-02T08:25:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,256 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import io.micrometer.core.instrument.MeterRegistry;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: remcowesterhoud
body: 👍 I like the test, very easy to read.
created_at: 2025-05-02T08:38:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: It only resets the (stateful) gauge values as they represent the db state, but doesn't touch the counters for example. If during replay an applier increments a counter, then they might be incremented multiple times per event (in case they were processed before and replayed again or multiple times due to restart loop or so). It also depends on the scraping cycle of prometheus and if during restart we somewhat clean up / reset the meter registry.
created_at: 2025-05-02T09:50:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -247,6 +289,7 @@ public CommandDistributionRecord getCommandDistributionRecord(
 
     return new CommandDistributionRecord()
         .setPartitionId(partition)
+        .setQueueId(persistedDistribution.getQueueId().orElse(""))
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: Ah k, will do. I just saw that empty string was defined as default value in the entity, thus I thought better set it like that.
created_at: 2025-05-02T09:51:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,6 +221,7 @@ private void startDistributing(
         commandDistributionDistributing
             .setPartitionId(partition)
             .setValueType(valueType)
+            .setQueueId(queueId)
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: eppdot
body: Yeah, it makes sense, but it's also hard to understand when you just look at a single piece of code and also prone to errors 😬 
created_at: 2025-05-02T09:53:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,7 +24,11 @@ public CommandDistributionDistributingApplier(final MutableDistributionState dis
 
   @Override
   public void applyState(final long key, final CommandDistributionRecord value) {
-    distributionState.addPendingDistribution(key, value.getPartitionId());
+    if (Strings.isEmpty(value.getQueueId())) {
+      // enqueued items are already added to the pending distributions
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingApplier.java
user: login: eppdot
body: Oh that's interesting. But that also means, we have to be super careful changing any of the methods that get called from an applier and methods that are getting called from there and so on 😄 
created_at: 2025-05-02T10:04:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,7 +24,11 @@ public CommandDistributionDistributingApplier(final MutableDistributionState dis
 
   @Override
   public void applyState(final long key, final CommandDistributionRecord value) {
-    distributionState.addPendingDistribution(key, value.getPartitionId());
+    if (Strings.isEmpty(value.getQueueId())) {
+      // enqueued items are already added to the pending distributions
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingApplier.java
user: login: eppdot
body: I am just thinking, if it wouldn't be better to do the refactoring that all distributions have a queue now 🤔 otherwise we would need then a 3rd version of the applier soonish again.
created_at: 2025-05-02T10:08:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).removePendingDistribution();
+  }
+
+  public void addInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addInflightDistribution();
+  }
+
+  public void removeInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).removeInflightDistribution();
+  }
+
+  /**
+   * This method is called on the target partition when a distribution command has been successfully
+   * processed and an acknowledgement is sent back to the origin partition.
+   *
+   * @param originPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   */
+  public void sentAcknowledgeDistribution(final int originPartitionId) {
+    getPartitionMetrics(originPartitionId).sentAcknowledgeDistribution();
+  }
+
+  /**
+   * This method is called on the origin partition when a distribution command has been successfully
+   * acknowledged by the target partition.
+   *
+   * @param targetPartitionId the source partition id of the distribution (where the distribution
+   *     was started)
+   */
+  public void receivedAcknowledgeDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).receivedAcknowledgeDistribution();
+  }
+
+  /**
+   * This method is called when a current inflight distribution is being retried. This may happen if
+   * the acknowledgement of the target partition is not received by the origin partition in-time.
+   * See {@link CommandRedistribution}.
+   *
+   * @param targetPartitionId the target partition id of the distribution
+   */
+  public void retryInflightDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).retryInflightDistribution();
+  }
+
+  private PartitionDistributionMetrics getPartitionMetrics(final int targetPartitionId) {
+    return partitionMetrics.computeIfAbsent(
+        targetPartitionId, id -> new PartitionDistributionMetrics(id, meterRegistry));
+  }
+
+  public void timedRedistribution(final Runnable redistribution) {
+    // TODO: wrap with a long task timer
+    redistribution.run();
+  }
+
+  private class PartitionDistributionMetrics {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: entangled90
body: :wrench: can it be static? otherwise the field `meterRegistry` is redundant as it is available from the outer class. I'd go with static if possible.
created_at: 2025-05-09T07:45:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: entangled90
body: :thought_balloon:  It would be interesting to have the type of record that is being redistributed, but adding this will probably generate too much metrics. Same for all other metrics
created_at: 2025-05-09T07:47:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: entangled90
body: :wrench: all these methods may be collapsed into a single one with an additional tag:
```java
public void addDistribution(final int partitionId, DistributionStatus status)
```

created_at: 2025-05-09T07:48:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.state.appliers;
+
+import io.camunda.zeebe.engine.state.TypedEventApplier;
+import io.camunda.zeebe.engine.state.mutable.MutableDistributionState;
+import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import org.agrona.Strings;
+
+public final class CommandDistributionDistributingV2Applier
+    implements TypedEventApplier<CommandDistributionIntent, CommandDistributionRecord> {
+
+  private final MutableDistributionState distributionState;
+
+  public CommandDistributionDistributingV2Applier(
+      final MutableDistributionState distributionState) {
+    this.distributionState = distributionState;
+  }
+
+  @Override
+  public void applyState(final long key, final CommandDistributionRecord value) {
+    if (Strings.isEmpty(value.getQueueId())) {
+      // added to pending state via {@link CommandDistributionEnqueuedApplier}
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+    distributionState.addRetriableDistribution(key, value.getPartitionId());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingV2Applier.java
user: login: entangled90
body: :question: is this change related to metrics? Could you explain why it's needed?
created_at: 2025-05-09T08:11:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: entangled90
body: Can we add a noop `DistributionMetrics` during replay and then swap with a real one when we start processing?
created_at: 2025-05-09T08:13:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: You tell me 😄 , but that helps only with the counters incremented multiple times because of replay. Still if I understand correctly, it maybe also happen that the events haven't been counted before 🤔 thus, I believe we can't get something super consistent here. 

I believe in this thread it's more about the gauge recovery than the counters (were we accept, that they might be slightly off sometimes).
created_at: 2025-05-09T08:29:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: entangled90
body: I was wondering if we could "initialize" from rocksDB the gauges :thinking: 
It's probably not a small feat of work though (assuming we have all the info there)
created_at: 2025-05-09T08:32:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: eppdot
body: That's what is happening here, right? Or do I misunderstand your point?

This transfers the count from rocksdb to the gauge meter
```
commandDistributionRecordColumnFamily.forEach(
        (distributionKey, record) -> getMetrics().addDistribution(distributionKey.getValue()));
```
created_at: 2025-05-09T08:36:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +113,32 @@ public DbDistributionState(
             transactionContext,
             continuationByQueueKey,
             new PersistedCommandDistribution());
+
+    metrics = new DistributionMetrics(zeebeDb.getMeterRegistry());
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/distribution/DbDistributionState.java
user: login: entangled90
body: yeah, sorry  :see_no_evil: 
created_at: 2025-05-09T08:50:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: I was thinking at some point to add the queueId as an additional tag, which is on a bit higher level than the record type. But I would see this as a follow up and maybe first see while using the new metrics if the need arises.
created_at: 2025-05-09T08:54:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.state.appliers;
+
+import io.camunda.zeebe.engine.state.TypedEventApplier;
+import io.camunda.zeebe.engine.state.mutable.MutableDistributionState;
+import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import org.agrona.Strings;
+
+public final class CommandDistributionDistributingV2Applier
+    implements TypedEventApplier<CommandDistributionIntent, CommandDistributionRecord> {
+
+  private final MutableDistributionState distributionState;
+
+  public CommandDistributionDistributingV2Applier(
+      final MutableDistributionState distributionState) {
+    this.distributionState = distributionState;
+  }
+
+  @Override
+  public void applyState(final long key, final CommandDistributionRecord value) {
+    if (Strings.isEmpty(value.getQueueId())) {
+      // added to pending state via {@link CommandDistributionEnqueuedApplier}
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+    distributionState.addRetriableDistribution(key, value.getPartitionId());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingV2Applier.java
user: login: eppdot
body: Yes, otherwise it would be counted twice for enqueued distributions (see comment & commit message)
created_at: 2025-05-09T08:57:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: 🤔 it would mean (alternatives)

- I need a switch/if-else/map to select the right metric to increment
- I need to have a field in the enum to identify the metric to increment (coupling the enum to metric names)
- I need to / could use the `DistributionStatus` as a tag in the metrics (but pending & inflight are very different, thus this would make higher level aggregations very strange to interpret)
created_at: 2025-05-09T09:04:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: eppdot
body: I tried playing around with your proposal, but I think it's not worth it for these 2 kinds (pending, inflight) tbh 🤔 
created_at: 2025-05-09T09:16:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.state.appliers;
+
+import io.camunda.zeebe.engine.state.TypedEventApplier;
+import io.camunda.zeebe.engine.state.mutable.MutableDistributionState;
+import io.camunda.zeebe.protocol.impl.record.value.distribution.CommandDistributionRecord;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import org.agrona.Strings;
+
+public final class CommandDistributionDistributingV2Applier
+    implements TypedEventApplier<CommandDistributionIntent, CommandDistributionRecord> {
+
+  private final MutableDistributionState distributionState;
+
+  public CommandDistributionDistributingV2Applier(
+      final MutableDistributionState distributionState) {
+    this.distributionState = distributionState;
+  }
+
+  @Override
+  public void applyState(final long key, final CommandDistributionRecord value) {
+    if (Strings.isEmpty(value.getQueueId())) {
+      // added to pending state via {@link CommandDistributionEnqueuedApplier}
+      distributionState.addPendingDistribution(key, value.getPartitionId());
+    }
+    distributionState.addRetriableDistribution(key, value.getPartitionId());
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/state/appliers/CommandDistributionDistributingV2Applier.java
user: login: entangled90
body: I guess it's for setting up the gauge correctly after recovering?
created_at: 2025-05-09T09:28:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,212 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import io.camunda.zeebe.util.micrometer.MicrometerUtil;
+import io.camunda.zeebe.util.micrometer.StatefulGauge;
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.MeterRegistry;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public final class DistributionMetrics {
+
+  private final MeterRegistry meterRegistry;
+  private final Map<Integer, PartitionDistributionMetrics> partitionMetrics =
+      new ConcurrentHashMap<>();
+
+  // Metrics
+  private final StatefulGauge activeDistributionsGauge;
+  private final Counter distributionsCounter;
+  private boolean isResetting = false;
+
+  public DistributionMetrics(final MeterRegistry meterRegistry) {
+    this.meterRegistry = meterRegistry;
+
+    activeDistributionsGauge =
+        StatefulGauge.builder(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.ACTIVE_COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+
+    distributionsCounter =
+        Counter.builder(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getName())
+            .description(DistributionMetricsDoc.COMMAND_DISTRIBUTIONS.getDescription())
+            .register(meterRegistry);
+  }
+
+  public MeterRegistry getMeterRegistry() {
+    return meterRegistry;
+  }
+
+  public void startReset() {
+    isResetting = true;
+    activeDistributionsGauge.set(0);
+    partitionMetrics.values().forEach(PartitionDistributionMetrics::resetGauges);
+  }
+
+  public void finalizeReset() {
+    isResetting = false;
+  }
+
+  public void addDistribution() {
+    if (!isResetting) {
+      distributionsCounter.increment();
+    }
+    activeDistributionsGauge.increment();
+  }
+
+  public void removeDistribution() {
+    activeDistributionsGauge.decrement();
+  }
+
+  public void addPendingDistribution(final int targetPartitionId) {
+    getPartitionMetrics(targetPartitionId).addPendingDistribution();
+  }
+
+  public void removePendingDistribution(final int targetPartitionId) {
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/metrics/DistributionMetrics.java
user: login: entangled90
body: I think it's worth it if we can do it for:
- active
- pending
- in-flight
- acknowledge

I'm not sure in-flight-retries fits into this :shrug: 
So we should create an enum in the MetricsDoc which will be used as values of the tag.
Then you can create an `EnumMap` containing all the gauges.
created_at: 2025-05-09T09:32:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,14 +585,10 @@ private static void addCommandDistributionProcessors(
       final CommandDistributionBehavior commandDistributionBehavior,
       final TypedRecordProcessors typedRecordProcessors,
       final Writers writers,
-      final ProcessingState processingState,
-      final Supplier<ScheduledTaskState> scheduledTaskStateFactory,
-      final InterPartitionCommandSender interPartitionCommandSender) {
+      final ProcessingState processingState) {
 
     // periodically retries command distribution
-    typedRecordProcessors.withListener(
-        new CommandRedistributor(
-            scheduledTaskStateFactory.get().getDistributionState(), interPartitionCommandSender));
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
user: login: eppdot
body: @korthout this is the change I am most unsure about 🤔 
created_at: 2025-05-20T10:28:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 🔧 This has a chance to be flaky. There might be a chance that the distribution is retried, e.g. when things are slow. Either, guarantee that we don't retry (disable the redistributor), or let's remove the zero retries assertion. 

Note that with a retry the ack sent/received might also be greater than one. 
Also note that this case exists in `shouldRepopulateGaugesAfterRecovery`
created_at: 2025-05-21T12:05:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: ❓ Shouldn't `count` be one?
created_at: 2025-05-21T12:10:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  private long triggerUnqueuedDistribution() {
+    final var reset = engine.clock().reset();
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 🔧 Let's clarify that this clock reset is a command that distributed in an unqueued manner, rather than a reset of the actor clock. This is useful to know because we add time to the actor clock when waiting for distribution retries. It's important to know that these are completely separate and don't interact with one-another.
created_at: 2025-05-21T12:14:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  private long triggerUnqueuedDistribution() {
+    final var reset = engine.clock().reset();
+    waitUntilCommandDistributionIs(CommandDistributionIntent.DISTRIBUTING, reset.getKey());
+    return reset.getKey();
+  }
+
+  private long triggerQueuedDistribution() {
+    final var deploy =
+        engine
+            .deployment()
+            .withXmlResource(
+                "process.bpmn", Bpmn.createExecutableProcess().startEvent().endEvent().done())
+            .expectCreated()
+            .deploy();
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 💭 This test assumes that the deployment is written to partition 1 (see `snapshotMetrics()`), but implicitly depends on this because that's how deployments are implemented currently. That might change in the future. I don't think we have to do something for this now, as `DeploymentClient` currently doesn't support a `.onPartition(int)` method or so, but wanted to share it for completeness.
created_at: 2025-05-21T12:28:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 👍 I really like the tests. Very easy to understand.
created_at: 2025-05-21T12:49:00Z
author_association: MEMBER
diff_hunk: @@ -79,13 +74,16 @@ public void onRecovered(final ReadonlyStreamProcessorContext context) {
 
   private void runRetryCycle() {
     final var retriableDistributions = new HashSet<RetriableDistribution>();
-    distributionState.foreachRetriableDistribution(
-        (distributionKey, record) -> {
-          final var retriable = new RetriableDistribution(distributionKey, record.getPartitionId());
-          retriableDistributions.add(retriable);
-          retryDistribution(retriable, record);
-          return true;
-        });
+    distributionBehavior
+        .getDistributionState()
+        .foreachRetriableDistribution(
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: korthout
body: ❌ Why do we get the state from the behavior class? That seems wrong to me, because I expect that the behavior class has the processingState instead of the scheduledTaskState. The redistributor (scheduled task) should not look in the processingState due to concurrency between the stream processor and the scheduled tasks.
created_at: 2025-05-21T13:24:54Z
author_association: MEMBER
diff_hunk: @@ -59,19 +62,49 @@ public final class CommandDistributionBehavior {
   private final CommandDistributionRecord commandDistributionContinuation =
       new CommandDistributionRecord();
 
+  private final DistributionMetrics distributionMetrics;
+
   public CommandDistributionBehavior(
       final DistributionState distributionState,
       final Writers writers,
       final int currentPartition,
       final RoutingInfo routingInfo,
-      final InterPartitionCommandSender partitionCommandSender) {
+      final InterPartitionCommandSender partitionCommandSender,
+      final DistributionMetrics distributionMetrics) {
     this.distributionState = distributionState;
     commandWriter = writers.command();
     stateWriter = writers.state();
     sideEffectWriter = writers.sideEffect();
     this.routingInfo = routingInfo;
     interPartitionCommandSender = partitionCommandSender;
     currentPartitionId = currentPartition;
+    this.distributionMetrics = distributionMetrics;
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    getMetrics().reset();
+
+    getDistributionState()
+        .foreachCommandDistribution(
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: korthout
body: 🔧 It can directly access `distributionState`, no need to call a public getter internally.

```suggestion
    distributionState
        .foreachCommandDistribution(
```

The same goes for access of the other fields (e.g. `distributionMetrics` as `getMetrics()`)
created_at: 2025-05-21T13:27:46Z
author_association: MEMBER
diff_hunk: @@ -586,14 +585,10 @@ private static void addCommandDistributionProcessors(
       final CommandDistributionBehavior commandDistributionBehavior,
       final TypedRecordProcessors typedRecordProcessors,
       final Writers writers,
-      final ProcessingState processingState,
-      final Supplier<ScheduledTaskState> scheduledTaskStateFactory,
-      final InterPartitionCommandSender interPartitionCommandSender) {
+      final ProcessingState processingState) {
 
     // periodically retries command distribution
-    typedRecordProcessors.withListener(
-        new CommandRedistributor(
-            scheduledTaskStateFactory.get().getDistributionState(), interPartitionCommandSender));
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
user: login: korthout
body: Yes, this is not safe IMO. It's important that the `CommandRedistributor` uses the state provided by `scheduledTaskStateFactory.get()`.
created_at: 2025-05-21T14:25:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: eppdot
body: No, for counters it's expected that they "reset" from time to time and we only recover gauges from state.
created_at: 2025-05-21T14:28:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 🔧 Can we document that with `.describedAs` to make it clear?
created_at: 2025-05-21T15:55:16Z
author_association: MEMBER
diff_hunk: @@ -586,14 +585,10 @@ private static void addCommandDistributionProcessors(
       final CommandDistributionBehavior commandDistributionBehavior,
       final TypedRecordProcessors typedRecordProcessors,
       final Writers writers,
-      final ProcessingState processingState,
-      final Supplier<ScheduledTaskState> scheduledTaskStateFactory,
-      final InterPartitionCommandSender interPartitionCommandSender) {
+      final ProcessingState processingState) {
 
     // periodically retries command distribution
-    typedRecordProcessors.withListener(
-        new CommandRedistributor(
-            scheduledTaskStateFactory.get().getDistributionState(), interPartitionCommandSender));
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/EngineProcessors.java
user: login: eppdot
body: Yeah that's what I thought as well. Thus if I want to follow the pattern to use the behavior in the redistributor, then I would need to create a new behavior, right?
created_at: 2025-05-22T06:08:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: eppdot
body: Okay, I am fine with removing the assertion as "tweaking" the test to do no retries at all is probably not worth it. But it also means we remove the "ack" assertions as well.
created_at: 2025-05-22T06:11:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  private long triggerUnqueuedDistribution() {
+    final var reset = engine.clock().reset();
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: eppdot
body: How would you propose to do this? a comment or do you have something specific in mind? otherwise the method is already called `triggerUnqueuedDistribution` 🤷
created_at: 2025-05-22T06:13:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,13 +74,16 @@ public void onRecovered(final ReadonlyStreamProcessorContext context) {
 
   private void runRetryCycle() {
     final var retriableDistributions = new HashSet<RetriableDistribution>();
-    distributionState.foreachRetriableDistribution(
-        (distributionKey, record) -> {
-          final var retriable = new RetriableDistribution(distributionKey, record.getPartitionId());
-          retriableDistributions.add(retriable);
-          retryDistribution(retriable, record);
-          return true;
-        });
+    distributionBehavior
+        .getDistributionState()
+        .foreachRetriableDistribution(
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: eppdot
body: Yep, that was the point from above and the "drafted" idea to also use the behavior within the redistributor. When we align on using the behavior (a newly created behavior) here as well, then I am happy continue the refactor 😄 (that's what I meant in slack)

I actually like using the behavior as it will encapsulate all relevant logic for command distribution within the behavior 🕵️ 
created_at: 2025-05-22T06:15:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -59,19 +62,49 @@ public final class CommandDistributionBehavior {
   private final CommandDistributionRecord commandDistributionContinuation =
       new CommandDistributionRecord();
 
+  private final DistributionMetrics distributionMetrics;
+
   public CommandDistributionBehavior(
       final DistributionState distributionState,
       final Writers writers,
       final int currentPartition,
       final RoutingInfo routingInfo,
-      final InterPartitionCommandSender partitionCommandSender) {
+      final InterPartitionCommandSender partitionCommandSender,
+      final DistributionMetrics distributionMetrics) {
     this.distributionState = distributionState;
     commandWriter = writers.command();
     stateWriter = writers.state();
     sideEffectWriter = writers.sideEffect();
     this.routingInfo = routingInfo;
     interPartitionCommandSender = partitionCommandSender;
     currentPartitionId = currentPartition;
+    this.distributionMetrics = distributionMetrics;
+  }
+
+  @Override
+  public void onRecovered(final ReadonlyStreamProcessorContext context) {
+    getMetrics().reset();
+
+    getDistributionState()
+        .foreachCommandDistribution(
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: eppdot
body: I actually liked having these methods and it also helped me to incrementally refactor and move the metrics from `DbDistributionState` to the behavior 🤔 
created_at: 2025-05-22T06:16:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,246 @@
+/*
+ * Copyright Camunda Services GmbH and/or licensed to Camunda Services GmbH under
+ * one or more contributor license agreements. See the NOTICE file distributed
+ * with this work for additional information regarding copyright ownership.
+ * Licensed under the Camunda License 1.0. You may not use this file
+ * except in compliance with the Camunda License 1.0.
+ */
+package io.camunda.zeebe.engine.metrics;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import io.camunda.zeebe.engine.processing.distribution.CommandRedistributor;
+import io.camunda.zeebe.engine.util.EngineRule;
+import io.camunda.zeebe.model.bpmn.Bpmn;
+import io.camunda.zeebe.protocol.record.RejectionType;
+import io.camunda.zeebe.protocol.record.intent.CommandDistributionIntent;
+import io.camunda.zeebe.test.util.record.RecordingExporter;
+import io.camunda.zeebe.test.util.record.RecordingExporterTestWatcher;
+import io.camunda.zeebe.util.collection.Tuple;
+import org.awaitility.Awaitility;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.RuleChain;
+import org.junit.rules.TestWatcher;
+
+public class CommandDistributionMetricsTest {
+
+  private final TestWatcher watcher = new RecordingExporterTestWatcher();
+  private final EngineRule engine = EngineRule.multiplePartition(2);
+
+  @Rule public final RuleChain ruleChain = RuleChain.outerRule(engine).around(watcher);
+
+  @Test
+  public void shouldTrackNormalDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne());
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  @Test
+  public void shouldTrackRetriedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    waitUntilDistributionIsRetried(key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    // wait for second (retried) acknowledgement to be processed
+    RecordingExporter.commandDistributionRecords(CommandDistributionIntent.ACKNOWLEDGE)
+        .withRejectionType(RejectionType.NOT_FOUND)
+        .withRecordKey(key)
+        .await();
+
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isGreaterThanOrEqualTo(1),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldTrackQueuedDistribution() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var distributions = new Tuple<>(triggerQueuedDistribution(), triggerQueuedDistribution());
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isEqualTo(2),
+            metrics -> assertThat(metrics.pending).isEqualTo(2),
+            metrics -> assertThat(metrics.inflight).isEqualTo(1));
+
+    // when
+    engine.resumeProcessing(2);
+
+    // then
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, distributions.getRight());
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isEqualTo(2),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isGreaterThanOrEqualTo(2),
+            metrics -> assertThat(metrics.ackReceived).isGreaterThanOrEqualTo(2));
+  }
+
+  @Test
+  public void shouldRepopulateGaugesAfterRecovery() {
+    // given
+    engine.pauseProcessing(2);
+
+    // when
+    final var key = triggerUnqueuedDistribution();
+
+    // then
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isOne(),
+            metrics -> assertThat(metrics.active).isOne(),
+            metrics -> assertThat(metrics.pending).isOne(),
+            metrics -> assertThat(metrics.inflight).isOne(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isZero(),
+            metrics -> assertThat(metrics.ackReceived).isZero());
+
+    // when
+    engine.stop();
+    engine.start(); // starts processing partition 2 again
+
+    // then
+    // we need to wait for processing to be done, to have deterministic assertions
+    // thus we can only assert that after recovery the gauges were repopulated to 1
+    // and then through the continued processing decremented back to zero
+    // (signaled by the ackSent/Received)
+    waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
+    assertThat(snapshotMetrics())
+        .satisfies(
+            metrics -> assertThat(metrics.count).isZero(),
+            metrics -> assertThat(metrics.active).isZero(),
+            metrics -> assertThat(metrics.pending).isZero(),
+            metrics -> assertThat(metrics.inflight).isZero(),
+            metrics -> assertThat(metrics.retries).isZero(),
+            metrics -> assertThat(metrics.ackSent).isOne(),
+            metrics -> assertThat(metrics.ackReceived).isOne());
+  }
+
+  private long triggerUnqueuedDistribution() {
+    final var reset = engine.clock().reset();
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: Yeah just an inline comment should do
created_at: 2025-05-22T09:36:29Z
author_association: MEMBER
diff_hunk: @@ -79,13 +74,16 @@ public void onRecovered(final ReadonlyStreamProcessorContext context) {
 
   private void runRetryCycle() {
     final var retriableDistributions = new HashSet<RetriableDistribution>();
-    distributionState.foreachRetriableDistribution(
-        (distributionKey, record) -> {
-          final var retriable = new RetriableDistribution(distributionKey, record.getPartitionId());
-          retriableDistributions.add(retriable);
-          retryDistribution(retriable, record);
-          return true;
-        });
+    distributionBehavior
+        .getDistributionState()
+        .foreachRetriableDistribution(
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandRedistributor.java
user: login: korthout
body: I understand, but this is problematic:
- the CommandRedistributor is a scheduled task and thus needs access to it's own ScheduledTaskState
- the CommandDistributionBehavior encapsulates the distribution behavior that is used by processors, so it has access to the ProcessingState
- :x: in these changes, we're providing the ProcessingState to the CommandRedistributor through the CommandDistributionBehavior

I expected that you would move some of the logic over to the behavior class, and create two instances of the behavior class (one with the ProcessingState and one with the ScheduledTaskState). But, that means that the caller needs to know which methods can be used when.

Note that the idea behind the behavior classes have always been to easily allow reusing some logic between processors. Scheduled tasks don't really add behavior, they just check something and write a command to the logstream or send a command to another partition. The trouble we encounter here shows that it's probably not a good idea to share the logic between the processors and the scheduled tasks.
created_at: 2025-05-22T09:50:13Z
author_association: MEMBER
diff_hunk: @@ -164,7 +161,11 @@ public void shouldRepopulateGaugesAfterRecovery() {
     waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
     assertThat(snapshotMetrics())
         .satisfies(
-            metrics -> assertThat(metrics.count).isZero(),
+            metrics ->
+                assertThat(metrics.count)
+                    .isZero()
+                    .describedAs(
+                        "Counters will not be recovered and are expected to reset to zero"),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 🔧 `describedAs` always comes before the assertion, otherwise it doesn't take affect
```suggestion
            metrics ->
                assertThat(metrics.count)
                    .describedAs("Counters will not be recovered and are expected to reset to zero")
                    .isZero(),
```
created_at: 2025-05-22T11:23:52Z
author_association: MEMBER
diff_hunk: @@ -174,6 +175,7 @@ public void shouldRepopulateGaugesAfterRecovery() {
   }
 
   private long triggerUnqueuedDistribution() {
+    // clock reset is used to trigger a command within the engine that requires distribution
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: korthout
body: 🔧 Can you add that this does not affect the actor clock, which is manipulated elsewhere in this class but that's unrelated
created_at: 2025-05-22T11:28:08Z
author_association: MEMBER
diff_hunk: @@ -59,19 +62,61 @@ public final class CommandDistributionBehavior {
   private final CommandDistributionRecord commandDistributionContinuation =
       new CommandDistributionRecord();
 
+  private final DistributionMetrics distributionMetrics;
+  private final Writers writers;
+
   public CommandDistributionBehavior(
       final DistributionState distributionState,
       final Writers writers,
       final int currentPartition,
       final RoutingInfo routingInfo,
-      final InterPartitionCommandSender partitionCommandSender) {
+      final InterPartitionCommandSender partitionCommandSender,
+      final DistributionMetrics distributionMetrics) {
     this.distributionState = distributionState;
+    this.writers = writers;
     commandWriter = writers.command();
     stateWriter = writers.state();
     sideEffectWriter = writers.sideEffect();
     this.routingInfo = routingInfo;
     interPartitionCommandSender = partitionCommandSender;
     currentPartitionId = currentPartition;
+    this.distributionMetrics = distributionMetrics;
+  }
+
+  public CommandDistributionBehavior withState(final DistributionState state) {
+    return new CommandDistributionBehavior(
+        state,
+        writers,
+        currentPartitionId,
+        routingInfo,
+        interPartitionCommandSender,
+        distributionMetrics);
+  }
path: zeebe/engine/src/main/java/io/camunda/zeebe/engine/processing/distribution/CommandDistributionBehavior.java
user: login: korthout
body: 💭 This method looks out of place, and highlights what I wrote about [here](https://github.com/camunda/camunda/pull/29804#discussion_r2102132773). At least we create a new instance which ensures a bit more threadsafety between the two actors.
created_at: 2025-05-22T12:08:31Z
author_association: MEMBER
diff_hunk: @@ -164,7 +161,11 @@ public void shouldRepopulateGaugesAfterRecovery() {
     waitUntilCommandDistributionIs(CommandDistributionIntent.FINISHED, key);
     assertThat(snapshotMetrics())
         .satisfies(
-            metrics -> assertThat(metrics.count).isZero(),
+            metrics ->
+                assertThat(metrics.count)
+                    .isZero()
+                    .describedAs(
+                        "Counters will not be recovered and are expected to reset to zero"),
path: zeebe/engine/src/test/java/io/camunda/zeebe/engine/metrics/CommandDistributionMetricsTest.java
user: login: eppdot
body: 👍 thanks, I didn't know that and the method was available after `.isZero()`. I also realized that some of the other tests had this potential flakyness, thus I changed/removed the retry/acknowledge checks.
created_at: 2025-05-23T05:49:25Z
author_association: CONTRIBUTOR

PR comments:
user: login: CLAassistant
created_at: 2025-04-07T13:00:24Z
body: [![CLA assistant check](https://cla-assistant.io/pull/badge/signed)](https://cla-assistant.io/camunda/camunda?pullRequest=29804) <br/>All committers have signed the CLA.
author_association: NONE
user: login: eppdot
created_at: 2025-05-23T06:54:59Z
body: >❓ Will you also add new Grafana panels?

Yes, that was the plan after this PR is merged. I don't know exactly when I will find time to tackle it, therefore I split the task into two to keep track of it.
- [32417](https://github.com/camunda/camunda/issues/32417) (will be closed with this PR)
- [32418](https://github.com/camunda/camunda/issues/32418)
author_association: CONTRIBUTOR
user: login: korthout
created_at: 2025-05-23T08:39:52Z
body: @eppdot It looks like there's a bug introduced with the partition id (not in the record value but the record metadata) of the FINISH command:

```
Error:  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.156 s <<< FAILURE! -- in io.camunda.zeebe.engine.processing.mapping.DeleteMappingMultiPartitionTest
Error:  io.camunda.zeebe.engine.processing.mapping.DeleteMappingMultiPartitionTest.shouldDistributeMappingDeleteCommand -- Time elapsed: 0.156 s <<< FAILURE!
java.lang.AssertionError: 

Expecting actual:
  [(CREATE, COMMAND, 1),
    (CREATED, EVENT, 1),
    (STARTED, EVENT, 1),
    (ENQUEUED, EVENT, 2),
    (DISTRIBUTING, EVENT, 2),
    (ENQUEUED, EVENT, 3),
    (DISTRIBUTING, EVENT, 3),
    (DELETE, COMMAND, 1),
    (ACKNOWLEDGE, COMMAND, 2),
    (ACKNOWLEDGE, COMMAND, 3),
    (DELETED, EVENT, 1),
    (STARTED, EVENT, 1),
    (ENQUEUED, EVENT, 2),
    (ENQUEUED, EVENT, 3),
    (ACKNOWLEDGED, EVENT, 2),
    (DISTRIBUTING, EVENT, 2),
    (ACKNOWLEDGE, COMMAND, 2),
    (ACKNOWLEDGED, EVENT, 3),
    (DISTRIBUTING, EVENT, 3),
    (FINISH, COMMAND, 3),
    (FINISHED, EVENT, 3),
    (ACKNOWLEDGED, EVENT, 2),
    (ACKNOWLEDGE, COMMAND, 3),
    (ACKNOWLEDGED, EVENT, 3),
    (FINISH, COMMAND, 3),
    (FINISHED, EVENT, 3)]
to end with:
  [(FINISHED, EVENT, 1)]
```
author_association: MEMBER
user: login: eppdot
created_at: 2025-05-23T09:12:56Z
body: > @eppdot It looks like there's a bug introduced with the partition id (not in the record value but the record metadata) of the FINISH command:
> 
> ```
> Error:  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.156 s <<< FAILURE! -- in io.camunda.zeebe.engine.processing.mapping.DeleteMappingMultiPartitionTest
> Error:  io.camunda.zeebe.engine.processing.mapping.DeleteMappingMultiPartitionTest.shouldDistributeMappingDeleteCommand -- Time elapsed: 0.156 s <<< FAILURE!
> java.lang.AssertionError: 
> 
> Expecting actual:
>   [(CREATE, COMMAND, 1),
>     (CREATED, EVENT, 1),
>     (STARTED, EVENT, 1),
>     (ENQUEUED, EVENT, 2),
>     (DISTRIBUTING, EVENT, 2),
>     (ENQUEUED, EVENT, 3),
>     (DISTRIBUTING, EVENT, 3),
>     (DELETE, COMMAND, 1),
>     (ACKNOWLEDGE, COMMAND, 2),
>     (ACKNOWLEDGE, COMMAND, 3),
>     (DELETED, EVENT, 1),
>     (STARTED, EVENT, 1),
>     (ENQUEUED, EVENT, 2),
>     (ENQUEUED, EVENT, 3),
>     (ACKNOWLEDGED, EVENT, 2),
>     (DISTRIBUTING, EVENT, 2),
>     (ACKNOWLEDGE, COMMAND, 2),
>     (ACKNOWLEDGED, EVENT, 3),
>     (DISTRIBUTING, EVENT, 3),
>     (FINISH, COMMAND, 3),
>     (FINISHED, EVENT, 3),
>     (ACKNOWLEDGED, EVENT, 2),
>     (ACKNOWLEDGE, COMMAND, 3),
>     (ACKNOWLEDGED, EVENT, 3),
>     (FINISH, COMMAND, 3),
>     (FINISHED, EVENT, 3)]
> to end with:
>   [(FINISHED, EVENT, 1)]
> ```


@korthout This issue was brought in here as `record.getPartitionId()` got replaced by `recordValue.getPartitionId()` with the false assumption that both would be the same. The logic moved into the behavior, thus I tried to keep the number of params provided low 😬 

```
final var finishRecord =
          new CommandDistributionRecord()
              .setPartitionId(record.getPartitionId())
              .setValueType(recordValue.getValueType())
              .setIntent(recordValue.getIntent());
      queueId.ifPresent(finishRecord::setQueueId);

      commandWriter.appendFollowUpCommand(
          distributionKey, CommandDistributionIntent.FINISH, finishRecord);
```

Would you say that changing it to use `currentPartitionId` in the behavior is good enough, or better pass down the partitionId from the original record? (or `final var originPartition = Protocol.decodePartitionId(distributionKey);`)
```
final var finishRecord =
          new CommandDistributionRecord()
              .setPartitionId(currentPartitionId)
              .setValueType(recordValue.getValueType())
              .setIntent(recordValue.getIntent());
      queueId.ifPresent(finishRecord::setQueueId);
```


author_association: CONTRIBUTOR