Repository: redpanda-data/redpanda
Pull request #25236: Pre-restart probe tests
Author: login: bashtanov
Labels: name: area/redpanda
Created: 2025-03-03T14:48:14Z | Merged: 2025-03-31T08:48:01Z

https://redpandadata.atlassian.net/browse/CORE-8663
Introduce failure injection toggling endpoint to force replica lag. Add tests for pre- and post-restart probes.

<!--
See https://github.com/redpanda-data/redpanda/blob/dev/CONTRIBUTING.md#pull-request-body
for more details and examples of what is expected in a PR body.

Content in this top section is REQUIRED. Describe, in plain language, the motivation
behind the change (bug fix, feature, improvement) in this PR and how the included
commits address it.

Add the GitHub keyword `Fixes` to link to bug(s) this PR will fix, e.g.
  Fixes #ISSUE-NUMBER, Fixes #ISSUE-NUMBER, ...

If this PR is a backport, link to the original with `Backport of PR`, e.g.
  Backport of PR #PR-NUMBER
-->

## Backports Required

<!-- Checking at least one of the checkboxes is REQUIRED if this PR is not a backport. -->

- [ ] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [x] v25.1.x
- [ ] v24.3.x
- [ ] v24.2.x
- [ ] v24.1.x

## Release Notes

* none
<!--
If the changes in this PR do not need to be mentioned in the release
notes, then don't add a sub-section and simply list `none`, e.g.

* none

Otherwise, adding a sub-section or `none` is REQUIRED if the PR is not a backport PR.
If this is a backport PR, adding contents to this section will override
the release notes section inherited from the original PR to dev.

Add one or more of the sub-sections with a short description bullet
point of the change, e.g.

### Bug Fixes

* Short description of the bug fix if this is a PR to `dev` branch.

### Features

* Short description of the feature. Explain how to configure.

### Improvements

* Short description of how this PR improves existing behavior.

-->



Diff:
diff --git a/src/v/cluster/tests/health_bench.cc b/src/v/cluster/tests/health_bench.cc
index 0a04313436dbc..66a7aa8157527 100644
--- a/src/v/cluster/tests/health_bench.cc
+++ b/src/v/cluster/tests/health_bench.cc
@@ -87,7 +87,8 @@ struct health_bench : health_report_accessor {
                     partition_status status{
                       .id{pid},
                       .leader_id = std::nullopt,
-                      .under_replicated_replicas = 1};
+                      .under_replicated_replicas = 1,
+                      .followers_stats = {{}}};
                     statuses.at(nid).partitions.emplace_back(std::move(status));
                 }
             }
diff --git a/src/v/cluster/tests/health_monitor_bench.cc b/src/v/cluster/tests/health_monitor_bench.cc
index 132e09234a9c8..9f0363fc25d8f 100644
--- a/src/v/cluster/tests/health_monitor_bench.cc
+++ b/src/v/cluster/tests/health_monitor_bench.cc
@@ -23,6 +23,18 @@ cluster::topic_status make_topic_status(size_t id, size_t num_partitions) {
         part_status.reclaimable_size_bytes = 100;
         part_status.revision_id = model::revision_id(1);
         part_status.term = model::term_id(1);
+        if (i % 3 == 0) { // leaders
+            if (i % 30) { // 10% of leaders to have faulty partitions
+                part_status.under_replicated_replicas.emplace(2);
+                part_status.followers_stats.emplace(cluster::followers_stats{
+                  .in_sync = 2,
+                  .out_of_sync = {model::node_id(1), model::node_id(2)},
+                  .down = {model::node_id(3), model::node_id(4)}});
+            } else {
+                part_status.under_replicated_replicas.emplace(0);
+                part_status.followers_stats.emplace();
+            }
+        }
 
         ts.partitions.push_back(part_status);
     }
diff --git a/src/v/raft/consensus.cc b/src/v/raft/consensus.cc
index 966a188f4f441..7d5924103d2fc 100644
--- a/src/v/raft/consensus.cc
+++ b/src/v/raft/consensus.cc
@@ -2246,6 +2246,16 @@ consensus::do_append_entries(append_entries_request&& r) {
             _hbeat = clock_type::now();
         });
         validate_offset_translator_delta(request_metadata, lstats);
+
+        // simulate disk error
+        if (unlikely(_inject_error_in_append_entries)) {
+            vlog(
+              _ctxlog.warn,
+              "simulating an error in append_entries request from {}",
+              r.source_node());
+            throw std::runtime_error("injected error");
+        }
+
         storage::append_result ofs = co_await disk_append(
           std::move(r).release_batches(), update_last_quorum_index::no);
         auto last_visible = std::min(
diff --git a/src/v/raft/consensus.h b/src/v/raft/consensus.h
index 4ea7b883d086c..a199894e10062 100644
--- a/src/v/raft/consensus.h
+++ b/src/v/raft/consensus.h
@@ -543,6 +543,15 @@ class consensus {
         return _configuration_manager.has_configuration_override();
     }
 
+    // start/stop simulating failed requests (for debug purposes)
+    void toggle_append_entries_error_injection(bool inject_error) {
+        vlog(
+          _ctxlog.warn,
+          "toggle_append_entries_error_injection block={}",
+          inject_error);
+        _inject_error_in_append_entries = inject_error;
+    }
+
 private:
     friend replication_monitor;
     friend replicate_entries_stm;
@@ -946,6 +955,9 @@ class consensus {
 
     replication_monitor _replication_monitor;
 
+    // simulate storage issues
+    bool _inject_error_in_append_entries = false;
+
     friend std::ostream& operator<<(std::ostream&, const consensus&);
 };
 
diff --git a/src/v/redpanda/admin/api-doc/debug.json b/src/v/redpanda/admin/api-doc/debug.json
index 575edb54d19ce..1cd4487fdb185 100644
--- a/src/v/redpanda/admin/api-doc/debug.json
+++ b/src/v/redpanda/admin/api-doc/debug.json
@@ -490,6 +490,74 @@
                 }
             ]
         },
+        {
+            "path": "/v1/debug/partitions/{namespace}/{topic}/{partition}/enable_error_injection/append_entries",
+            "operations": [
+                {
+                    "method": "POST",
+                    "summary": "Enable error simulation on raft append_entries request on this node",
+                    "type": "void",
+                    "nickname": "enable_append_entries_error_injection",
+                    "produces": [
+                        "application/json"
+                    ],
+                    "parameters": [
+                        {
+                            "name": "namespace",
+                            "in": "path",
+                            "required": true,
+                            "type": "string"
+                        },
+                        {
+                            "name": "topic",
+                            "in": "path",
+                            "required": true,
+                            "type": "string"
+                        },
+                        {
+                            "name": "partition",
+                            "in": "path",
+                            "required": true,
+                            "type": "integer"
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "path": "/v1/debug/partitions/{namespace}/{topic}/{partition}/disable_error_injection/append_entries",
+            "operations": [
+                {
+                    "method": "POST",
+                    "summary": "Disable error simulation on raft append_entries request on this node",
+                    "type": "void",
+                    "nickname": "disable_append_entries_error_injection",
+                    "produces": [
+                        "application/json"
+                    ],
+                    "parameters": [
+                        {
+                            "name": "namespace",
+                            "in": "path",
+                            "required": true,
+                            "type": "string"
+                        },
+                        {
+                            "name": "topic",
+                            "in": "path",
+                            "required": true,
+                            "type": "string"
+                        },
+                        {
+                            "name": "partition",
+                            "in": "path",
+                            "required": true,
+                            "type": "integer"
+                        }
+                    ]
+                }
+            ]
+        },
         {
             "path": "/v1/debug/unsafe_reset_metadata/{topic}/{partition}",
             "operations": [
diff --git a/src/v/redpanda/admin/partition.cc b/src/v/redpanda/admin/partition.cc
index f06b992b89b0d..abf556e84c467 100644
--- a/src/v/redpanda/admin/partition.cc
+++ b/src/v/redpanda/admin/partition.cc
@@ -536,6 +536,31 @@ admin_server::force_set_partition_replicas_handler(
     co_return ss::json::json_void();
 }
 
+ss::future<ss::json::json_return_type>
+admin_server::toggle_append_entries_error_injection(
+  std::unique_ptr<ss::http::request> req, bool inject) {
+    auto ntp = parse_ntp_from_request(req->param);
+    auto shard = _shard_table.local().shard_for(ntp);
+    if (!shard) {
+        throw ss::httpd::bad_request_exception(fmt_with_ctx(
+          fmt::format, "Partition {} not found on this node", ntp));
+    }
+
+    co_return co_await _partition_manager.invoke_on(
+      *shard,
+      [ntp = std::move(ntp), inject](cluster::partition_manager& pm) mutable
+      -> ss::future<ss::json::json_return_type> {
+          auto partition = pm.get(ntp);
+          if (!partition) {
+              return ss::make_exception_future<ss::json::json_return_type>(
+                ss::httpd::bad_request_exception(fmt_with_ctx(
+                  fmt::format, "Partition {} not found on this node", ntp)));
+          }
+          partition->raft()->toggle_append_entries_error_injection(inject);
+          return ssx::now<ss::json::json_return_type>(ss::json::json_void());
+      });
+}
+
 ss::future<ss::json::json_return_type>
 admin_server::set_partition_replicas_handler(
   std::unique_ptr<ss::http::request> req) {
@@ -591,7 +616,6 @@ admin_server::set_partition_replicas_handler(
 }
 
 namespace {
-
 json::validator make_set_replica_core_validator() {
     const std::string schema = R"(
 {
@@ -804,6 +828,18 @@ void admin_server::register_partition_routes() {
           return force_set_partition_replicas_handler(std::move(req));
       });
 
+    register_route<superuser>(
+      ss::httpd::debug_json::enable_append_entries_error_injection,
+      [this](std::unique_ptr<ss::http::request> req) {
+          return toggle_append_entries_error_injection(std::move(req), true);
+      });
+
+    register_route<superuser>(
+      ss::httpd::debug_json::disable_append_entries_error_injection,
+      [this](std::unique_ptr<ss::http::request> req) {
+          return toggle_append_entries_error_injection(std::move(req), false);
+      });
+
     register_route<superuser>(
       ss::httpd::partition_json::trigger_partitions_rebalance,
       [this](std::unique_ptr<ss::http::request> req) {
@@ -1089,7 +1125,6 @@ admin_server::get_majority_lost_partitions(
 }
 
 namespace {
-
 json::validator make_node_id_array_validator() {
     const std::string schema = R"(
     {
diff --git a/src/v/redpanda/admin/server.h b/src/v/redpanda/admin/server.h
index 9ca971df6cc95..5def4493700f5 100644
--- a/src/v/redpanda/admin/server.h
+++ b/src/v/redpanda/admin/server.h
@@ -541,6 +541,9 @@ class admin_server {
       set_partition_replicas_handler(std::unique_ptr<ss::http::request>);
     ss::future<ss::json::json_return_type>
       force_set_partition_replicas_handler(std::unique_ptr<ss::http::request>);
+    ss::future<ss::json::json_return_type>
+    toggle_append_entries_error_injection(
+      std::unique_ptr<ss::http::request>, bool inject);
     ss::future<ss::json::json_return_type>
       set_partition_replica_core_handler(std::unique_ptr<ss::http::request>);
 
diff --git a/tests/rptest/services/admin.py b/tests/rptest/services/admin.py
index b03ee7f7ef031..06db01ed00ed5 100644
--- a/tests/rptest/services/admin.py
+++ b/tests/rptest/services/admin.py
@@ -1190,6 +1190,21 @@ def force_set_partition_replicas(self,
         path = f"debug/partitions/{namespace}/{topic}/{partition}/force_replicas"
         return self._request('post', path, node=node, json=replicas)
 
+    def toggle_failure_injection(
+        self,
+        topic,
+        partition,
+        op,
+        *,
+        inject: bool,
+        node,
+        namespace="kafka",
+    ):
+        assert op == "append_entries"
+        verb = "enable" if inject else "disable"
+        path = f"debug/partitions/{namespace}/{topic}/{partition}/{verb}_error_injection/{op}"
+        return self._request('post', path, node=node)
+
     def cancel_partition_move(self,
                               topic,
                               partition,
diff --git a/tests/rptest/tests/node_restart_probe_test.py b/tests/rptest/tests/node_restart_probe_test.py
new file mode 100644
index 0000000000000..e254f1c034aed
--- /dev/null
+++ b/tests/rptest/tests/node_restart_probe_test.py
@@ -0,0 +1,444 @@
+# Copyright 2022 Redpanda Data, Inc.
+#
+# Use of this software is governed by the Business Source License
+# included in the file licenses/BSL.md
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0
+
+from contextlib import contextmanager
+from typing import Dict, Iterable, Sequence, Tuple
+import abc
+import random
+import re
+
+from rptest.clients.types import TopicSpec
+from rptest.services.admin import Admin
+from rptest.services.cluster import cluster
+from rptest.services.redpanda import RESTART_LOG_ALLOW_LIST
+from rptest.tests.redpanda_test import RedpandaTest
+
+from ducktape.cluster.cluster import ClusterNode
+from ducktape.utils.util import wait_until
+from rptest.clients.kafka_cli_tools import KafkaCliTools
+
+
+class Risks(dict):
+    """a Dict[str, frozenset[str]] that makes sure keys are exactly these: """
+    KEYS = frozenset(('rf1_offline', 'full_acks_produce_unavailable',
+                      'unavailable', 'acks1_data_loss'))
+    VALUE_RE = '^kafka/'  # ignore system topics
+
+    @classmethod
+    def build_value(cls, input: Iterable[str]):
+        return frozenset(v for v in input if re.match(cls.VALUE_RE, v))
+
+    def __init__(self, **kvargs):
+        keys = kvargs.keys()
+        assert keys == self.KEYS, f"{keys=}, {self.KEYS=}"
+        dict.__init__(self, **{
+            k: self.build_value(v)
+            for k, v in kvargs.items()
+        })
+
+
+NO_RISKS = Risks(**{typ: set() for typ in Risks.KEYS})
+
+
+class NodeRestartProbeTestBase(RedpandaTest):
+    MSG_SIZE = 1024
+
+    def __init__(self, *args, **kwargs):
+        super(NodeRestartProbeTestBase, self).__init__(*args, **kwargs)
+        self.admin = Admin(self.redpanda)
+        self.kafka_tools = KafkaCliTools(self.redpanda)
+
+    @contextmanager
+    def with_append_entries_error_injection(self: RedpandaTest,
+                                            node: ClusterNode,
+                                            partitions: Sequence[Tuple[str,
+                                                                       int]]):
+        node_id = self.redpanda.node_id(node)
+
+        def toggle(inject: bool):
+            for topic, partition in partitions:
+                self.redpanda.logger.info(
+                    "toggle append_entries failure injection "
+                    f"{topic=} {partition=} {node_id=} {inject=}")
+                self.admin.toggle_failure_injection(topic,
+                                                    partition,
+                                                    "append_entries",
+                                                    inject=inject,
+                                                    node=node)
+
+        toggle(True)
+        try:
+            yield
+        finally:
+            toggle(False)
+
+    def produce_to_all_partitions(self, acks):
+        for topic in self.topics:
+            self.redpanda.logger.debug(f"producing to {topic.name}")
+            produce_bytes = (self.PRODUCE_BYTES +
+                             self.PRODUCE_BYTES_JITTER * random.uniform(-1, 1))
+            produce_messages = max(produce_bytes / self.MSG_SIZE, 1)
+            num_messages = int(topic.partition_count * produce_messages)
+            self.kafka_tools.produce(topic.name, num_messages, self.MSG_SIZE,
+                                     acks)
+            self.redpanda.logger.debug(f"produced to {topic.name}")
+
+    @abc.abstractmethod
+    def create_topics(self):
+        """ should populate self.topics too """
+
+
+class NodePreRestartProbeTest(NodeRestartProbeTestBase):
+    PRODUCE_BYTES = 300 * 1024 * 1024
+    PRODUCE_BYTES_JITTER = 0
+
+    def __init__(self, test_context):
+        super(NodePreRestartProbeTest, self).__init__(
+            test_context=test_context,
+            num_brokers=5,
+            extra_rp_conf={
+                'health_monitor_max_metadata_age': 100,  # ms
+                'enable_leader_balancer': False
+            })
+
+    def create_topics(self):
+        self.topics = [
+            TopicSpec(name='t1', partition_count=1, replication_factor=1),
+            TopicSpec(name='t3', partition_count=2, replication_factor=3),
+            TopicSpec(name='t5', partition_count=1, replication_factor=5),
+        ]
+        self.client().create_topic_with_assignment(self.topics[0].name, [[1]])
+        self.client().create_topic_with_assignment(self.topics[1].name,
+                                                   [[1, 2, 3], [3, 4, 5]])
+        self.client().create_topic(self.topics[2])
+
+    def get_node_risks(self, node, limit=None) -> Risks:
+        reply = self.admin.get_broker_pre_restart_probe(node=node, limit=limit)
+        self.redpanda.logger.debug(f"get_risks returned: {reply}")
+        return Risks(**reply['risks'])
+
+    def get_risks(self) -> Dict[int, Risks]:
+        return {
+            self.redpanda.node_id(node): self.get_node_risks(node)
+            for node in self.redpanda.started_nodes()
+        }
+
+    def wait_pre_restart_probes(self,
+                                expected_risks: Dict[int, Risks],
+                                timeout_sec=30):
+        """wait until it returns expected result, make sure it 
+        does not return anything milder in the meanwhile"""
+        def risks_are_as_expected():
+            actual_risks = self.get_risks()
+            self.redpanda.logger.debug(
+                f"actual_risks={sorted(actual_risks.items())}, "
+                f"expected_risks={sorted(expected_risks.items())}")
+            return actual_risks == expected_risks
+
+        wait_until(risks_are_as_expected,
+                   timeout_sec=timeout_sec,
+                   backoff_sec=0.1,
+                   err_msg="Waiting for reported risks to match expected")
+
+    @cluster(num_nodes=5, log_allow_list=RESTART_LOG_ALLOW_LIST)
+    def pre_restart_probe_test(self):
+        nodes = {
+            self.redpanda.node_id(node): node
+            for node in self.redpanda.nodes
+        }
+
+        self.create_topics()
+        t1 = self.topics[0].name
+        t3 = self.topics[1].name
+        t5 = self.topics[2].name
+        t1p = f"kafka/{t1}/0"
+        t3p0 = f"kafka/{t3}/0"
+        t3p1 = f"kafka/{t3}/1"
+        t5p = f"kafka/{t5}/0"
+
+        # all nodes up
+        inevitable_risks = {
+            1:
+            Risks(rf1_offline=[t1p],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            2:
+            NO_RISKS,
+            3:
+            NO_RISKS,
+            4:
+            NO_RISKS,
+            5:
+            NO_RISKS,
+        }
+        self.wait_pre_restart_probes(inevitable_risks)
+        # limit 0 cuts off
+        assert self.get_node_risks(nodes[1], limit=0) == NO_RISKS
+
+        self.redpanda.stop_node(nodes[3])
+
+        # node 3 down
+        self.wait_pre_restart_probes({
+            1:
+            Risks(rf1_offline=[t1p],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[t3p0],
+                  acks1_data_loss=[]),
+            2:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[t3p0],
+                  acks1_data_loss=[]),
+            4:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[t3p1],
+                  acks1_data_loss=[]),
+            5:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[t3p1],
+                  acks1_data_loss=[]),
+        })
+
+        self.redpanda.start_node(nodes[3])
+        # move t3_0, t3_1 and t5 leaders off node 3 which we will make lagged
+        assert self.admin.transfer_leadership_to(namespace="kafka",
+                                                 topic=t3,
+                                                 partition=0,
+                                                 target_id=2)
+        assert self.admin.transfer_leadership_to(namespace="kafka",
+                                                 topic=t3,
+                                                 partition=1,
+                                                 target_id=4)
+        assert self.admin.transfer_leadership_to(namespace="kafka",
+                                                 topic=t5,
+                                                 partition=0,
+                                                 target_id=2)
+        self.redpanda.stop_node(nodes[5])
+
+        # lag node 3
+        with self.with_append_entries_error_injection(nodes[3], [(t3, 0),
+                                                                 (t3, 1),
+                                                                 (t5, 0)]):
+            self.produce_to_all_partitions(acks=1)
+
+        # node 3 lags, node 5 down
+        self.wait_pre_restart_probes({
+            1:
+            Risks(rf1_offline=[t1p],
+                  full_acks_produce_unavailable=[t3p0, t5p],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            2:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t3p0, t5p],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            3:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[],
+                  unavailable=[t3p1],
+                  acks1_data_loss=[]),
+            4:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t5p],
+                  unavailable=[t3p1],
+                  acks1_data_loss=[t3p1]),
+        })
+        # good time to see how limits work
+        assert len(
+            self.get_node_risks(nodes[1],
+                                limit=0)["full_acks_produce_unavailable"]) == 0
+        assert len(
+            self.get_node_risks(nodes[1],
+                                limit=1)["full_acks_produce_unavailable"]) == 1
+        assert len(
+            self.get_node_risks(nodes[1],
+                                limit=2)["full_acks_produce_unavailable"]) == 2
+        assert len(
+            self.get_node_risks(nodes[1],
+                                limit=3)["full_acks_produce_unavailable"]) == 2
+
+        # move t3_1 and t5 leaders off nodes 3 and 5 which we will make lagged
+        assert self.admin.transfer_leadership_to(namespace="kafka",
+                                                 topic=t3,
+                                                 partition=1,
+                                                 target_id=4)
+        assert self.admin.transfer_leadership_to(namespace="kafka",
+                                                 topic=t5,
+                                                 partition=0,
+                                                 target_id=2)
+        # lag nodes 3 and 5
+        self.redpanda.start_node(nodes[5])
+        with self.with_append_entries_error_injection(nodes[3], [
+            (t3, 0), (t3, 1), (t5, 0)
+        ]), self.with_append_entries_error_injection(nodes[5], [(t3, 1),
+                                                                (t5, 0)]):
+            self.produce_to_all_partitions(acks=1)
+
+        # all nodes up, but 3 and 5 lag
+        self.wait_pre_restart_probes({
+            1:
+            Risks(rf1_offline=[t1p],
+                  full_acks_produce_unavailable=[t3p0, t5p],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            2:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t3p0, t5p],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            3:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t3p1],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+            4:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t5p],
+                  unavailable=[],
+                  acks1_data_loss=[t3p1]),
+            5:
+            Risks(rf1_offline=[],
+                  full_acks_produce_unavailable=[t3p1],
+                  unavailable=[],
+                  acks1_data_loss=[]),
+        })
+
+        # when lag clears
+        self.wait_pre_restart_probes(inevitable_risks, timeout_sec=240)
+
+
+# Same as `wait_until(lambda: value_fn() == target, **kwargs)`, but also checks
+# that:
+# 1) intermediate values can be increased by no more than `max_drop` each but
+# no higher than to `target` to form a non-decreasing sequence.
+# 2) it receives at least `min_values` values, apart from `bottom` and `target`
+def wait_gradually_increases(value_fn, target, max_drop, min_values, **kwargs):
+    max_seen_value = None
+    distinct_values = set()
+
+    def completed():
+        nonlocal max_seen_value, distinct_values
+        cur_value = value_fn()
+        if max_seen_value is None:
+            max_seen_value = cur_value
+        assert cur_value >= max_seen_value - max_drop, (
+            f"received {cur_value} after {max_seen_value}")
+        max_seen_value = max(max_seen_value, cur_value)
+        if len(distinct_values) < min_values:
+            distinct_values.add(cur_value)
+        return cur_value == target and len(distinct_values) == min_values
+
+    wait_until(completed, **kwargs)
+
+
+def unittest_wait_gradually_increases():
+    def make_val_fn(*values):
+        it = iter(values)
+        return lambda: next(it)
+
+    default_params = dict(target=100,
+                          max_drop=2,
+                          min_values=5,
+                          timeout_sec=1,
+                          backoff_sec=0)
+
+    def call_wgi(*values):
+        wait_gradually_increases(make_val_fn(*values), **default_params)
+
+    def expect_wgi_pass(*values):
+        call_wgi(*values)
+
+    def expect_wgi_fail(*values):
+        try:
+            call_wgi(*values)
+        except StopIteration:
+            pass
+        except AssertionError:
+            pass
+        else:
+            assert False, "should have failed"
+
+    expect_wgi_pass(1, 0, 60, 58, 100)
+    expect_wgi_pass(1, 0, 50, 48, 78, 100)
+    expect_wgi_fail(5, 60, 58, 58, 60, 100)  # too few distinct vals
+    expect_wgi_fail(11, 40, 60, 58, 56, 100)  # decreases too much
+    expect_wgi_fail(1, 40, 60, 58, 59, 99)  # does not reach 100
+
+
+class NodePostRestartProbeTest(NodeRestartProbeTestBase):
+    PRODUCE_BYTES = 1 * 1024 * 1024
+    PRODUCE_BYTES_JITTER = 1 * 1024 * 1024
+
+    def __init__(self, test_context):
+        super(NodePostRestartProbeTest, self).__init__(
+            test_context=test_context,
+            num_brokers=3,
+            extra_rp_conf={'health_monitor_max_metadata_age': 100})
+
+    def create_topics(self):
+        self.topics = [
+            TopicSpec(partition_count=10, replication_factor=3)
+            for _ in range(10)
+        ]
+        self.client().create_topic(self.topics)
+
+    def get_load_reclaimed_pc(self, node):
+        load_reclamed_pc = self.admin.get_broker_post_restart_probe(
+            node)["load_reclaimed_pc"]
+        assert 0 <= load_reclamed_pc <= 100
+        self.redpanda.logger.info(f"{load_reclamed_pc=}")
+        return load_reclamed_pc
+
+    @cluster(num_nodes=3, log_allow_list=RESTART_LOG_ALLOW_LIST)
+    def post_restart_probe_test(self):
+        unittest_wait_gradually_increases()
+
+        self.create_topics()
+
+        lagging_node = random.choice(self.redpanda.nodes)
+
+        self.produce_to_all_partitions(acks=1)
+
+        wait_until(
+            lambda: self.get_load_reclaimed_pc(lagging_node) == 100,
+            timeout_sec=10,
+            backoff_sec=1,
+            err_msg="non-lagged replica load_reclamed_pc won't reach 100%")
+
+        all_partitions = [(t.name, pid) for t in self.topics
+                          for pid in range(t.partition_count)]
+        with self.with_append_entries_error_injection(lagging_node,
+                                                      all_partitions):
+            self.produce_to_all_partitions(acks=1)
+
+        # system partitions won't lag, some data partitions catch up quickly
+        wait_until(lambda: self.get_load_reclaimed_pc(lagging_node) <= 75,
+                   timeout_sec=10,
+                   backoff_sec=0.1,
+                   err_msg="lagged replica load_reclamed_pc won't go down")
+
+        wait_gradually_increases(
+            lambda: self.get_load_reclaimed_pc(lagging_node),
+            target=100,
+            max_drop=10,
+            min_values=3,
+            timeout_sec=30,
+            backoff_sec=0.1,
+            err_msg="lagged replica load_reclamed_pc won't reach 100%")
+
+        for n in self.redpanda.nodes:
+            wait_until(
+                lambda: self.get_load_reclaimed_pc(n) == 100,
+                timeout_sec=10,
+                backoff_sec=2,
+                err_msg="non-lagged replica load_reclamed_pc won't reach 100%")


Reviews:
user: login: mmaslankaprv
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T12:37:46Z
user: login: mmaslankaprv
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T12:39:09Z
user: login: bashtanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T12:43:12Z
user: login: bashtanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T12:45:09Z
user: login: mmaslankaprv
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-31T08:45:47Z

Review comments:
diff_hunk: @@ -2246,6 +2246,17 @@ consensus::do_append_entries(append_entries_request&& r) {
             _hbeat = clock_type::now();
         });
         validate_offset_translator_delta(request_metadata, lstats);
+
+        // simulate disk error
+        if (unlikely(_append_entries_blocked)) {
+            vlog(
+              _ctxlog.warn,
+              "ignoring append_entries request from {}: blocked",
+              r.source_node());
+            reply.result = reply_result::failure;
+            co_return reply;
path: src/v/raft/consensus.cc
user: login: mmaslankaprv
body: I am still not sure if that is the right way of doing failure injection, this is still doing a polite thing i.e. it sets failure and returns a reply. Can we throw an exception here ?
created_at: 2025-03-11T12:37:46Z
author_association: MEMBER
diff_hunk: @@ -490,6 +490,86 @@
                 }
             ]
         },
+        {
+            "path": "/v1/debug/partitions/{namespace}/{topic}/{partition}/block_raft_op/{op}",
path: src/v/redpanda/admin/api-doc/debug.json
user: login: mmaslankaprv
body: I do not particularly like the name `block_raft_of`. This is still very specific operation that has nothing to do with the actual failure that may happen in production system
created_at: 2025-03-11T12:39:09Z
author_association: MEMBER
diff_hunk: @@ -2246,6 +2246,17 @@ consensus::do_append_entries(append_entries_request&& r) {
             _hbeat = clock_type::now();
         });
         validate_offset_translator_delta(request_metadata, lstats);
+
+        // simulate disk error
+        if (unlikely(_append_entries_blocked)) {
+            vlog(
+              _ctxlog.warn,
+              "ignoring append_entries request from {}: blocked",
+              r.source_node());
+            reply.result = reply_result::failure;
+            co_return reply;
path: src/v/raft/consensus.cc
user: login: bashtanov
body: Sure I'll change it. It's not going to change the logic, as we are in a try block that will catch the exception and reply the same. It'll be correct w.r.t. future logic changes this way.
created_at: 2025-03-11T12:43:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -490,6 +490,86 @@
                 }
             ]
         },
+        {
+            "path": "/v1/debug/partitions/{namespace}/{topic}/{partition}/block_raft_op/{op}",
path: src/v/redpanda/admin/api-doc/debug.json
user: login: bashtanov
body: `append_entries_error_injection` with values `on` and `off`?
created_at: 2025-03-11T12:45:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: bashtanov
created_at: 2025-03-04T18:07:10Z
body: @mmaslankaprv how do you like this placement of error injection?
author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-03-04T22:07:52Z
body: <!--TEST RESULTS-->
<!--e93ce0d5-857c-4088-8562-f1d3444d5d9f-->
<!--TEST RESULTS-->
<!--f4956cff-1361-44cf-8de4-b6e25d09b3a4-->
<!--title-->
## CI test results
<!--/title-->
<!--body-->
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62608">build#62608</a></summary>


| test_id | test_kind | job_url | test_status | passed |
|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.full_disk_test.LogStorageMaxSizeSI.test_stay_below_target_size.log_segment_size=1048576.cleanup_policy=compact | ducktape | https://buildkite.com/redpanda/redpanda/builds/62608#019562b4-0ad6-4b60-a9f6-20ddf221a648 | FLAKY | 1/2 |
| rptest.tests.partition_balancer_test.PartitionBalancerTest.test_unavailable_nodes | ducktape | https://buildkite.com/redpanda/redpanda/builds/62608#019562a5-9888-4caa-b9ea-2e4d5d90b38c | FLAKY | 1/2 |
| rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=True.with_iceberg=False.with_chunked_compaction=True.cloud_storage_type=CloudStorageType.ABS | ducktape | https://buildkite.com/redpanda/redpanda/builds/62608#019562a5-9888-4caa-b9ea-2e4d5d90b38c | FLAKY | 1/2 |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62987">build#62987</a></summary>


| test_id                                                                                               | test_kind   | job_url                                                                                   | test_status   | passed   |
|:------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62987#01958b40-e0af-4b95-8fd2-b79804544ffb | FLAKY         | 1/2      |
| rptest.tests.node_restart_probe_test.NodePostRestartProbeTest.post_restart_probe_test                 | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62987#01958b40-e0b0-4f6e-9858-114203adc4fe | FLAKY         | 1/2      |
| rptest.tests.node_restart_probe_test.NodePreRestartProbeTest.pre_restart_probe_test                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62987#01958b3d-b954-40fa-bccb-0063ba095fd4 | FLAKY         | 1/2      |
</details>
<!--/body-->
author_association: COLLABORATOR
user: login: mmaslankaprv
created_at: 2025-03-11T12:40:24Z
body: I am still not convinced. The solution is still not generic enough to be considered a failure injector.
author_association: MEMBER
user: login: vbotbuildovich
created_at: 2025-03-31T08:48:23Z
body: /backport v25.1.x
author_association: COLLABORATOR