Repository: redpanda-data/redpanda
Pull request #25649: datalake/coordinator: group duplicate ensure_table calls
Author: login: andrwng
Labels: name: area/build
name: area/redpanda
Created: 2025-03-29T16:15:22Z | Merged: 2025-03-30T17:18:36Z

<!--
See https://github.com/redpanda-data/redpanda/blob/dev/CONTRIBUTING.md#pull-request-body
for more details and examples of what is expected in a PR body.

Content in this top section is REQUIRED. Describe, in plain language, the motivation
behind the change (bug fix, feature, improvement) in this PR and how the included
commits address it.

Add the GitHub keyword `Fixes` to link to bug(s) this PR will fix, e.g.
  Fixes #ISSUE-NUMBER, Fixes #ISSUE-NUMBER, ...

If this PR is a backport, link to the original with `Backport of PR`, e.g.
  Backport of PR #PR-NUMBER
-->
In certain scenarios with a large number of partitions I would see that
sometimes translation would get stuck, with RPCs to the coordinator all
timing out. I suspect that this is because of the large volume of
ensure_table_schema requests that could come in, particularly right
after a restart, when all translators are starting up together.

To mitigate this, this commit adds tracking of on-going requests to
ensure_table_schema and groups incoming requests in with on-going ones,
if they exist.

## Backports Required

<!-- Checking at least one of the checkboxes is REQUIRED if this PR is not a backport. -->

- [ ] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [x] v25.1.x
- [ ] v24.3.x
- [ ] v24.2.x
- [ ] v24.1.x

## Release Notes

* none
<!--
If the changes in this PR do not need to be mentioned in the release
notes, then don't add a sub-section and simply list `none`, e.g.

* none

Otherwise, adding a sub-section or `none` is REQUIRED if the PR is not a backport PR.
If this is a backport PR, adding contents to this section will override
the release notes section inherited from the original PR to dev.

Add one or more of the sub-sections with a short description bullet
point of the change, e.g.

### Bug Fixes

* Short description of the bug fix if this is a PR to `dev` branch.

### Features

* Short description of the feature. Explain how to configure.

### Improvements

* Short description of how this PR improves existing behavior.

-->



Diff:
diff --git a/src/v/datalake/coordinator/BUILD b/src/v/datalake/coordinator/BUILD
index fce7ce140e8d6..d10d8426a9115 100644
--- a/src/v/datalake/coordinator/BUILD
+++ b/src/v/datalake/coordinator/BUILD
@@ -61,6 +61,7 @@ redpanda_cc_library(
         "//src/v/config",
         "//src/v/container:fragmented_vector",
         "//src/v/model",
+        "@abseil-cpp//absl/hash",
         "@seastar",
     ],
 )
diff --git a/src/v/datalake/coordinator/coordinator.cc b/src/v/datalake/coordinator/coordinator.cc
index c6a3ffd761e3b..0bff2972e8cdb 100644
--- a/src/v/datalake/coordinator/coordinator.cc
+++ b/src/v/datalake/coordinator/coordinator.cc
@@ -47,6 +47,39 @@ coordinator::errc convert_stm_errc(coordinator_stm::errc e) {
 }
 } // namespace
 
+coordinator::ensure_table_result_t coordinator::notify_waiters_and_erase(
+  const coordinator::table_key& key,
+  coordinator::ensure_table_map_t& in_flight_map,
+  ensure_table_result_t res) {
+    auto it = in_flight_map.find(key);
+    if (it == in_flight_map.end()) {
+        // Unexpected, but benign.
+        return res;
+    }
+    for (auto& prom : it->second) {
+        prom.set_value(res);
+    }
+    in_flight_map.erase(it);
+    return res;
+}
+
+std::optional<ss::future<coordinator::ensure_table_result_t>>
+coordinator::maybe_add_waiter(
+  const table_key& key, coordinator::ensure_table_map_t& in_flight_map) {
+    auto it = in_flight_map.find(key);
+    if (it != in_flight_map.end()) {
+        // There's an in-flight request already. Attach to it and wait for it
+        // to complete.
+        auto& proms = it->second;
+        ss::promise<ensure_table_result_t> res_prom;
+        auto res_fut = res_prom.get_future();
+        proms.emplace_back(std::move(res_prom));
+        return std::move(res_fut);
+    }
+    in_flight_map.insert({key, {}});
+    return std::nullopt;
+}
+
 std::ostream& operator<<(std::ostream& o, coordinator::errc e) {
     switch (e) {
     case coordinator::errc::not_leader:
@@ -295,8 +328,6 @@ coordinator::do_ensure_table_exists(
         co_return convert_stm_errc(sync_res.error());
     }
 
-    // TODO: add mutex to protect against the thundering herd problem
-
     auto topic_md = topic_table_.get_topic_metadata_ref(
       model::topic_namespace_view{model::kafka_namespace, topic});
     if (!topic_md || topic_md->get().get_revision() != topic_revision) {
@@ -424,12 +455,29 @@ coordinator::sync_ensure_table_exists(
   model::topic topic,
   model::revision_id topic_revision,
   record_schema_components comps) {
-    co_return co_await do_ensure_table_exists(
+    // If there's already an inflight request for this table, attach it to the
+    // in-flight request.
+    table_key key{
+      .topic = topic, .topic_revision = topic_revision, .record_comps = comps};
+    auto waiter_fut = maybe_add_waiter(key, in_flight_main_);
+    if (waiter_fut.has_value()) {
+        co_return co_await std::move(*waiter_fut);
+    }
+    auto res_fut = co_await ss::coroutine::as_future(do_ensure_table_exists(
       topic,
       topic_revision,
       std::move(comps),
       "sync_ensure_table_exists",
-      main_table_schema_provider{*this});
+      main_table_schema_provider{*this}));
+
+    if (res_fut.failed()) {
+        // NOTE: we don't expect any exceptions given we're using result types,
+        // but be conservative to guarantee fulfilling the promises.
+        auto ex = res_fut.get_exception();
+        vlog(datalake_log.error, "Exception ensuring table: {}", ex);
+        co_return notify_waiters_and_erase(key, in_flight_main_, errc::failed);
+    }
+    co_return notify_waiters_and_erase(key, in_flight_main_, res_fut.get());
 }
 
 struct coordinator::dlq_table_schema_provider
@@ -457,12 +505,27 @@ struct coordinator::dlq_table_schema_provider
 ss::future<checked<std::nullopt_t, coordinator::errc>>
 coordinator::sync_ensure_dlq_table_exists(
   model::topic topic, model::revision_id topic_revision) {
-    co_return co_await do_ensure_table_exists(
+    table_key key{
+      .topic = topic, .topic_revision = topic_revision, .record_comps = {}};
+    auto waiter_fut = maybe_add_waiter(key, in_flight_dlq_);
+    if (waiter_fut.has_value()) {
+        co_return co_await std::move(*waiter_fut);
+    }
+    auto res_fut = co_await ss::coroutine::as_future(do_ensure_table_exists(
       topic,
       topic_revision,
       record_schema_components{},
       "sync_ensure_dlq_table_exists",
-      dlq_table_schema_provider{*this});
+      dlq_table_schema_provider{*this}));
+
+    if (res_fut.failed()) {
+        // NOTE: we don't expect any exceptions given we're using result types,
+        // but be conservative to guarantee fulfilling the promises.
+        auto ex = res_fut.get_exception();
+        vlog(datalake_log.error, "Exception ensuring DLQ table: {}", ex);
+        co_return notify_waiters_and_erase(key, in_flight_dlq_, errc::failed);
+    }
+    co_return notify_waiters_and_erase(key, in_flight_dlq_, res_fut.get());
 }
 
 ss::future<checked<std::nullopt_t, coordinator::errc>>
diff --git a/src/v/datalake/coordinator/coordinator.h b/src/v/datalake/coordinator/coordinator.h
index 5ca8dc64098c4..00bf3250c2c2e 100644
--- a/src/v/datalake/coordinator/coordinator.h
+++ b/src/v/datalake/coordinator/coordinator.h
@@ -18,6 +18,8 @@
 #include "datalake/fwd.h"
 #include "model/fundamental.h"
 
+#include <absl/hash/hash.h>
+
 namespace datalake::coordinator {
 
 // Public interface that provides access to the coordinator STM. Conceptually,
@@ -88,6 +90,32 @@ class coordinator {
     bool leader_loop_running() const { return term_as_.has_value(); }
 
 private:
+    // Key used to deduplicate requests for the same table.
+    struct table_key {
+        model::topic topic;
+        model::revision_id topic_revision;
+        record_schema_components record_comps;
+        bool operator==(const table_key&) const = default;
+        template<typename H>
+        friend H AbslHashValue(H h, const table_key& k) {
+            return H::combine(
+              std::move(h), k.topic, k.topic_revision, k.record_comps);
+        }
+    };
+    // Functions to help avoid piling up duplicate requests for the same table
+    // key. The caller adds a waiter to an existing list of waiters if one
+    // exists, in which case they are expected to wait on the returned future.
+    // Otherwise, the caller may proceed immediately to make the request and
+    // _must_ notify the waiters upon finishing.
+    using ensure_table_result_t = checked<std::nullopt_t, errc>;
+    using ensure_table_map_t = chunked_hash_map<
+      table_key,
+      chunked_vector<ss::promise<ensure_table_result_t>>>;
+    static std::optional<ss::future<ensure_table_result_t>>
+    maybe_add_waiter(const table_key&, ensure_table_map_t&);
+    static ensure_table_result_t notify_waiters_and_erase(
+      const table_key&, ensure_table_map_t&, ensure_table_result_t);
+
     checked<ss::gate::holder, errc> maybe_gate();
 
     // Waits for leadership, and then reconciles STM state with external state
@@ -132,6 +160,9 @@ class coordinator {
     // Abort source that can be used to stop work in a given term.
     // Is only set if there is an on-going call to run_until_term_change().
     std::optional<std::reference_wrapper<ss::abort_source>> term_as_;
+
+    ensure_table_map_t in_flight_main_;
+    ensure_table_map_t in_flight_dlq_;
 };
 std::ostream& operator<<(std::ostream&, coordinator::errc);
 
diff --git a/src/v/iceberg/rest_catalog.cc b/src/v/iceberg/rest_catalog.cc
index b194591f59d3b..f0a761ab050fd 100644
--- a/src/v/iceberg/rest_catalog.cc
+++ b/src/v/iceberg/rest_catalog.cc
@@ -98,7 +98,8 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
   const table_identifier& t_id,
   const schema& schema,
   const partition_spec& spec) {
-    auto rtc = create_rtc();
+    auto parent_rtc = create_rtc();
+    auto table_rtc = retry_chain_node(&parent_rtc);
     vlog(log.trace, "create table {} requested", t_id);
     create_table_request request{
       .name = t_id.table,
@@ -106,7 +107,8 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
       .partition_spec = spec.copy()};
     auto h = co_await lock_.get_units();
 
-    auto res = co_await client_->create_table(t_id.ns, std::move(request), rtc);
+    auto res = co_await client_->create_table(
+      t_id.ns, std::move(request), table_rtc);
     if (res.has_value()) {
         co_return get_metadata(std::move(res.value()));
     }
@@ -121,8 +123,9 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
     create_namespace_request ns_request{
       .ns = t_id.ns.copy(),
     };
+    auto namespace_rtc = retry_chain_node(&parent_rtc);
     auto ns_res = co_await client_->create_namespace(
-      std::move(ns_request), rtc);
+      std::move(ns_request), namespace_rtc);
     if (!ns_res.has_value()) {
         auto ns_errc = map_error("create_namespace", ns_res.error());
         if (ns_errc != catalog::errc::already_exists) {
@@ -135,8 +138,9 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
       .name = t_id.table,
       .schema = schema.copy(),
       .partition_spec = spec.copy()};
-    co_return (
-      co_await client_->create_table(t_id.ns, std::move(retry_request), rtc))
+    auto table_retry_rtc = retry_chain_node(&parent_rtc);
+    co_return (co_await client_->create_table(
+                 t_id.ns, std::move(retry_request), table_retry_rtc))
       .transform(get_metadata)
       .transform_error([](const rest_client::domain_error& err) {
           return map_error("create_table_retry", err);
diff --git a/src/v/iceberg/rest_client/catalog_client.cc b/src/v/iceberg/rest_client/catalog_client.cc
index 280e7c5b3cc32..af64e4481cc64 100644
--- a/src/v/iceberg/rest_client/catalog_client.cc
+++ b/src/v/iceberg/rest_client/catalog_client.cc
@@ -114,9 +114,9 @@ catalog_client::catalog_client(
 
 ss::future<expected<std::monostate>>
 catalog_client::maybe_configure(retry_chain_node& rtc) {
-    auto gh = _gate.hold();
-    if (_configured) {
-        co_return std::monostate{};
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
     }
     vlog(log.debug, "Configuring Iceberg REST catalog client");
     auto http_request = http::request_builder{}
@@ -173,6 +173,13 @@ catalog_client::maybe_configure(retry_chain_node& rtc) {
     co_return std::monostate{};
 }
 
+expected<ss::gate::holder> catalog_client::maybe_gate() {
+    if (_gate.is_closed()) {
+        return tl::unexpected(aborted_error("catalog_client::_gate closed"));
+    }
+    return _gate.hold();
+}
+
 ss::future<expected<oauth_token>>
 catalog_client::acquire_token(retry_chain_node& rtc) {
     vassert(
@@ -264,7 +271,7 @@ ss::future<expected<std::monostate>> catalog_client::maybe_add_bearer_auth(
 }
 
 ss::future<expected<iobuf>> catalog_client::perform_request(
-  retry_chain_node& rtc,
+  retry_chain_node& parent_rtc,
   http::request_builder request_builder,
   const ss::sstring& host,
   client_probe::endpoint endpoint,
@@ -272,7 +279,7 @@ ss::future<expected<iobuf>> catalog_client::perform_request(
     if (payload.has_value()) {
         request_builder.with_content_length(payload.value().size_bytes());
     }
-
+    retry_chain_node rtc(&parent_rtc);
     std::vector<http_call_error> retriable_errors{};
 
     while (true) {
@@ -361,7 +368,10 @@ ss::future<expected<iobuf>> catalog_client::perform_request(
 ss::future<expected<create_namespace_response>>
 catalog_client::create_namespace(
   create_namespace_request req, retry_chain_node& rtc) {
-    auto gh = _gate.hold();
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
+    }
     auto config_result = co_await maybe_configure(rtc);
     if (!config_result.has_value()) {
         co_return tl::unexpected(config_result.error());
@@ -389,7 +399,10 @@ ss::future<expected<load_table_result>> catalog_client::create_table(
   const chunked_vector<ss::sstring>& ns,
   create_table_request req,
   retry_chain_node& rtc) {
-    auto gh = _gate.hold();
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
+    }
     auto config_result = co_await maybe_configure(rtc);
     if (!config_result.has_value()) {
         co_return tl::unexpected(config_result.error());
@@ -416,7 +429,10 @@ ss::future<expected<load_table_result>> catalog_client::load_table(
   const chunked_vector<ss::sstring>& ns,
   const ss::sstring& table_name,
   retry_chain_node& rtc) {
-    auto gh = _gate.hold();
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
+    }
     auto config_result = co_await maybe_configure(rtc);
     if (!config_result.has_value()) {
         co_return tl::unexpected(config_result.error());
@@ -440,7 +456,10 @@ ss::future<expected<std::monostate>> catalog_client::drop_table(
   const ss::sstring& table_name,
   std::optional<bool> purge_requested,
   retry_chain_node& rtc) {
-    auto gh = _gate.hold();
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
+    }
     auto config_result = co_await maybe_configure(rtc);
     if (!config_result.has_value()) {
         co_return tl::unexpected(config_result.error());
@@ -470,7 +489,10 @@ ss::future<expected<std::monostate>> catalog_client::drop_table(
 
 ss::future<expected<commit_table_response>> catalog_client::commit_table_update(
   commit_table_request commit_request, retry_chain_node& rtc) {
-    auto gh = _gate.hold();
+    auto gh = maybe_gate();
+    if (!gh.has_value()) {
+        co_return tl::unexpected(gh.error());
+    }
     auto config_result = co_await maybe_configure(rtc);
     if (!config_result.has_value()) {
         co_return tl::unexpected(config_result.error());
diff --git a/src/v/iceberg/rest_client/catalog_client.h b/src/v/iceberg/rest_client/catalog_client.h
index 2cceea920d51d..e7b250de14876 100644
--- a/src/v/iceberg/rest_client/catalog_client.h
+++ b/src/v/iceberg/rest_client/catalog_client.h
@@ -175,6 +175,8 @@ class catalog_client {
     ss::future<expected<std::monostate>> maybe_configure(retry_chain_node&);
 
 private:
+    expected<ss::gate::holder> maybe_gate();
+
     // The root url calculated from base url, prefix and api version. Given a
     // base url of "/b", an api version "v2" and a prefix of "x/y", the root url
     // is "/b/v2/x/y/". The root url is prefixed before rest entities used when
@@ -200,7 +202,7 @@ class catalog_client {
     // the request with optional payload, and takes care of retrying according
     // to policy
     ss::future<expected<iobuf>> perform_request(
-      retry_chain_node& rtc,
+      retry_chain_node& parent_rtc,
       http::request_builder request_builder,
       const ss::sstring& host,
       client_probe::endpoint endpoint,


Reviews:
user: login: WillemKauf
body: `notify_waiters_and_erase()` and `maybe_add_waiter()` feel like nice generics to have for use with any request type- do you think we will want to extend this pattern to future request types beyond `ensure_table_result_t`? Or is this the only "problematic" request type we are constantly hammering the catalog with?
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-29T16:53:38Z
user: login: andrwng
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-29T17:54:37Z
user: login: WillemKauf
body: Would it be beneficial/easy to add some sort of fixture test for `maybe_add_waiter()` and `notify_waiters_and_erase()`?

Not a blocker, just wondering. I assume you've already seen empirical results which show the change working/reducing stress on the coordinator
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-29T18:04:24Z
user: login: dotnwat
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-30T03:32:26Z
user: login: WillemKauf
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T16:26:24Z
user: login: WillemKauf
body: CI looks good :+1: 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T16:27:43Z
user: login: andrwng
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T17:18:14Z
user: login: WillemKauf
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-30T17:27:43Z

Review comments:
diff_hunk: @@ -88,6 +90,46 @@ class coordinator {
     bool leader_loop_running() const { return term_as_.has_value(); }
 
 private:
+    // Key used to deduplicate requests for the same table.
+    struct table_key {
+        model::topic topic;
+        model::revision_id topic_revision;
+        record_schema_components record_comps;
+        bool operator==(const table_key&) const = default;
+        template<typename H>
+        friend H AbslHashValue(H h, const table_key& k) {
+            return H::combine(
+              std::move(h), k.topic, k.topic_revision, k.record_comps);
+        }
+    };
+    struct table_key_hash {
path: src/v/datalake/coordinator/coordinator.h
user: login: WillemKauf
body: `table_key_hash` and `table_key_eq` are currently unused - are these meant to be used in the `using` typedef for `ensure_table_map_t`?
created_at: 2025-03-29T16:40:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -88,6 +90,46 @@ class coordinator {
     bool leader_loop_running() const { return term_as_.has_value(); }
 
 private:
+    // Key used to deduplicate requests for the same table.
+    struct table_key {
+        model::topic topic;
+        model::revision_id topic_revision;
+        record_schema_components record_comps;
+        bool operator==(const table_key&) const = default;
+        template<typename H>
+        friend H AbslHashValue(H h, const table_key& k) {
+            return H::combine(
+              std::move(h), k.topic, k.topic_revision, k.record_comps);
+        }
+    };
+    struct table_key_hash {
path: src/v/datalake/coordinator/coordinator.h
user: login: andrwng
body: Unneeded copy pasta, nice catch!
created_at: 2025-03-29T17:54:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +172,13 @@ catalog_client::maybe_configure(retry_chain_node& rtc) {
     co_return std::monostate{};
 }
 
+expected<ss::gate::holder> catalog_client::maybe_gate() {
+    if (_gate.is_closed()) {
+        return tl::unexpected(aborted_error("catalog_client::_gate closed"));
+    }
+    return _gate.hold();
+}
path: src/v/iceberg/rest_client/catalog_client.cc
user: login: dotnwat
body: neat
created_at: 2025-03-30T03:27:41Z
author_association: MEMBER
diff_hunk: @@ -98,15 +98,17 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
   const table_identifier& t_id,
   const schema& schema,
   const partition_spec& spec) {
-    auto rtc = create_rtc();
+    auto parent_rtc = create_rtc();
+    auto table_rtc = retry_chain_node(&parent_rtc);
path: src/v/iceberg/rest_catalog.cc
user: login: WillemKauf
body: nit: I wonder if these child `rtc`s could be more tightly scoped if we had a function like `retry_chain_node::make_child()` which would basically return the result of `retry_chain_node::retry_chain_node(retry_chain_node* parent)`- then the change would only be 

```cpp
auto res = co_await client_->create_table(
      t_id.ns, std::move(request), rtc.make_child());
```
created_at: 2025-03-30T16:26:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,15 +98,17 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
   const table_identifier& t_id,
   const schema& schema,
   const partition_spec& spec) {
-    auto rtc = create_rtc();
+    auto parent_rtc = create_rtc();
+    auto table_rtc = retry_chain_node(&parent_rtc);
path: src/v/iceberg/rest_catalog.cc
user: login: andrwng
body: Yea i think that makes sense. Seems like a missing part of the API that makes it easy for these kinds of bugs to happen, cc @lazin
created_at: 2025-03-30T17:18:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,15 +98,17 @@ ss::future<checked<table_metadata, catalog::errc>> rest_catalog::create_table(
   const table_identifier& t_id,
   const schema& schema,
   const partition_spec& spec) {
-    auto rtc = create_rtc();
+    auto parent_rtc = create_rtc();
+    auto table_rtc = retry_chain_node(&parent_rtc);
path: src/v/iceberg/rest_catalog.cc
user: login: WillemKauf
body: I guess part of the reason this doesn't already exist is the [existing laundry list of `retry_chain_node` constructors which take in a `retry_chain_node* parent`](https://github.com/redpanda-data/redpanda/blob/438336f27aeabac766990366fc8693271eb008ea/src/v/utils/retry_chain_node.h#L234-L265) which `make_child()` (and it's potential overloads) could decide to support :melting_face:  
created_at: 2025-03-30T17:27:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: andrwng
created_at: 2025-03-29T18:06:22Z
body: > `notify_waiters_and_erase()` and `maybe_add_waiter()` feel like nice generics to have for use with any request type- do you think we will want to extend this pattern to future request types beyond `ensure_table_result_t`? Or is this the only "problematic" request type we are constantly hammering the catalog with?

Yea something generic makes sense. We have a utils/waiter_queue that's already somewhat similar. As for other RPCs, I think we're good for now, the ensure_table RPCs are the only ones that end up immediately hitting the catalog.

There's probably room in the schema manager on the translators to have similar logic when loading the table from the catalog. Will consider that for a follow up.
author_association: CONTRIBUTOR
user: login: andrwng
created_at: 2025-03-29T18:07:58Z
body: > Would it be beneficial/easy to add some sort of fixture test for `maybe_add_waiter()` and `notify_waiters_and_erase()`?
> 
> Not a blocker, just wondering. I assume you've already seen empirical results which show the change working/reducing stress on the coordinator

Yea, makes sense. Will see about this on Monday.
author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-03-29T21:44:07Z
body: <!--TEST RESULTS-->
<!--591dbfd3-7bec-4d23-8bb4-b135725a42d2-->
<!--TEST RESULTS-->
<!--35a6c8e8-68ec-41f9-93f3-099a87bd6792-->
<!--title-->
## CI test results
<!--/title-->
<!--body-->
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/63931">build#63931</a></summary>


| test_id | test_kind | job_url | test_status | passed |
|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_avro_schema.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39e0-4dce-be7a-360327cbddb3 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_no_collision.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db6-41af-86d8-0e73c88b3943 | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_select_fails.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7dba-40ab-8a93-16a83691ae20 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_dropped_column_select_fails.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_illegal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=illegalpromotionint-.string.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39e0-4dce-be7a-360327cbddb3 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_illegal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=illegalpromotionint-.string.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7dba-40ab-8a93-16a83691ae20 | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_illegal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=illegalpromotionint-.string.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39de-453d-82d4-3a16b115c3cb | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=add_column.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39dd-4e99-a130-34c3374e28cd | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=drop_column.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db6-41af-86d8-0e73c88b3943 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=promote_column.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39dd-4e99-a130-34c3374e28cd | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=promote_column.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39e0-4dce-be7a-360327cbddb3 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=promote_column.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db6-41af-86d8-0e73c88b3943 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=promote_column.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39df-4052-8964-8c873b741e7c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.test_case=drop_column.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39e0-4dce-be7a-360327cbddb3 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.test_case=drop_column.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db6-41af-86d8-0e73c88b3943 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_legal_schema_evolution.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.test_case=promote_column.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_old_schema_writer.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=add_column.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_old_schema_writer.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.test_case=drop_column.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39df-4052-8964-8c873b741e7c | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_old_schema_writer.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.test_case=reorder_columns.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39de-453d-82d4-3a16b115c3cb | FLAKY | 1/3 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_old_schema_writer.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.test_case=reorder_columns.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_reorder_columns.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e34b-39df-4052-8964-8c873b741e7c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_reorder_columns.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.produce_mode=ProducerType.PROTO3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db8-4c21-94b1-1ba4443eea6c | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_reorder_columns.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.AVRO.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7db6-41af-86d8-0e73c88b3943 | FLAKY | 1/2 |
| rptest.tests.datalake.schema_evolution_test.SchemaEvolutionE2ETests.test_reorder_columns.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.produce_mode=ProducerType.PROTO2.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/63931#0195e350-7dba-40ab-8a93-16a83691ae20 | FLAKY | 1/2 |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/63944">build#63944</a></summary>


| test_id                                                                                                                                          | test_kind   | job_url                                                                                   | test_status   | passed   |
|:-------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=compact.delete | ducktape    | https://buildkite.com/redpanda/redpanda/builds/63944#0195e5f5-d733-4213-aac1-596e1561b2ce | FLAKY         | 1/2      |
| rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic                                                                  | ducktape    | https://buildkite.com/redpanda/redpanda/builds/63944#0195e5f5-d736-4544-9364-dbaa37d28656 | FLAKY         | 1/2      |
</details>
<!--/body-->
author_association: COLLABORATOR
user: login: WillemKauf
created_at: 2025-03-29T21:46:27Z
body: Lot of flakes in CI, probably a bug somewhere?
author_association: CONTRIBUTOR
user: login: andrwng
created_at: 2025-03-30T04:49:36Z
body: Moved to draft while debugging flakiness
author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-03-30T17:18:53Z
body: /backport v25.1.x
author_association: COLLABORATOR