Repository: redpanda-data/redpanda
Pull request #25310: dl/translator: add batching to translator
Author: login: bharathv
Labels: name: area/build
name: area/redpanda
Created: 2025-03-10T19:13:38Z | Merged: 2025-03-12T17:52:49Z

Adds batching to the partition translator. Files are not rolled once they hit 32 MiB limit or the lag window ends (whichever happens first). 

Future improvements:

* A dummy (periodic) lag tracker is introduced, pending integration with lag heuristic in a separate PR.
* Batching limit is hardcoded to 32 MiB to keep concurrent used bytes by iceberg low (until space management is fully integrated). Lower default keeps less pressure on space mgmt to enforce retention rules on segment data.
* Currently batching limit applies to all flushed bytes from a single partition (across all data writers, hence all iceberg partitions), can probably improved to enforce it on a per file basis.
*  Row group size is determined by the data flushed within a single scheduler iteration (30s), we can optimize the row group size further (larger row groups) by improving wait_for_data_to_translate() to wait for more data to mark itself ready for scheduling, this way larger chunk of data is flushed in a single iteration.

<!--
See https://github.com/redpanda-data/redpanda/blob/dev/CONTRIBUTING.md#pull-request-body
for more details and examples of what is expected in a PR body.

Content in this top section is REQUIRED. Describe, in plain language, the motivation
behind the change (bug fix, feature, improvement) in this PR and how the included
commits address it.

Add the GitHub keyword `Fixes` to link to bug(s) this PR will fix, e.g.
  Fixes #ISSUE-NUMBER, Fixes #ISSUE-NUMBER, ...

If this PR is a backport, link to the original with `Backport of PR`, e.g.
  Backport of PR #PR-NUMBER
-->

## Backports Required

<!-- Checking at least one of the checkboxes is REQUIRED if this PR is not a backport. -->

- [x] none - not a bug fix
- [ ] none - this is a backport
- [ ] none - issue does not exist in previous branches
- [ ] none - papercut/not impactful enough to backport
- [ ] v24.3.x
- [ ] v24.2.x
- [ ] v24.1.x

## Release Notes
* none

<!--
If the changes in this PR do not need to be mentioned in the release
notes, then don't add a sub-section and simply list `none`, e.g.

* none

Otherwise, adding a sub-section or `none` is REQUIRED if the PR is not a backport PR.
If this is a backport PR, adding contents to this section will override
the release notes section inherited from the original PR to dev.

Add one or more of the sub-sections with a short description bullet
point of the change, e.g.

### Bug Fixes

* Short description of the bug fix if this is a PR to `dev` branch.

### Features

* Short description of the feature. Explain how to configure.

### Improvements

* Short description of how this PR improves existing behavior.

-->



Diff:
diff --git a/src/v/datalake/datalake_manager.cc b/src/v/datalake/datalake_manager.cc
index e4210976eb462..8e602828eeccd 100644
--- a/src/v/datalake/datalake_manager.cc
+++ b/src/v/datalake/datalake_manager.cc
@@ -353,12 +353,16 @@ datalake_manager::handle_translator_state_change(const model::ntp& ntp) {
         _topic_table,
         _features,
         get_or_create_probe(partition->ntp()));
+    auto lag_tracker
+      = translation::translation_lag_tracker::make_default_lag_tracker(
+        partition, _topic_table->local());
 
     auto translator = std::make_unique<translation::partition_translator>(
       _sg,
       std::move(coordinator),
       std::move(data_src),
       std::move(translation_ctx),
+      std::move(lag_tracker),
       simple_time_jitter<ss::lowres_clock, std::chrono::milliseconds>{
         translation_jitter_base, translation_jitter},
       retry_max_timeout,
diff --git a/src/v/datalake/record_multiplexer.cc b/src/v/datalake/record_multiplexer.cc
index 5360aff401cea..a9c4c7ca5e9dc 100644
--- a/src/v/datalake/record_multiplexer.cc
+++ b/src/v/datalake/record_multiplexer.cc
@@ -354,6 +354,11 @@ size_t record_multiplexer::flushed_bytes() const {
     return result;
 }
 
+std::optional<kafka::offset>
+record_multiplexer::last_translated_offset() const {
+    return _result ? std::make_optional(_result->last_offset) : std::nullopt;
+}
+
 ss::future<result<std::nullopt_t, writer_error>>
 record_multiplexer::handle_invalid_record(
   translation_probe::invalid_record_cause cause,
diff --git a/src/v/datalake/record_multiplexer.h b/src/v/datalake/record_multiplexer.h
index 9c511c911ffba..6995be8ff98ba 100644
--- a/src/v/datalake/record_multiplexer.h
+++ b/src/v/datalake/record_multiplexer.h
@@ -107,6 +107,8 @@ class record_multiplexer {
 
     size_t flushed_bytes() const;
 
+    std::optional<kafka::offset> last_translated_offset() const;
+
 private:
     // Handles the given record components of a record that is invalid for the
     // target table.
diff --git a/src/v/datalake/tests/translator_test.cc b/src/v/datalake/tests/translator_test.cc
index 55134e4968373..4bf7f2531af87 100644
--- a/src/v/datalake/tests/translator_test.cc
+++ b/src/v/datalake/tests/translator_test.cc
@@ -26,8 +26,7 @@ class PartitionTranslatorTestFixture
             add_node();
         }
         co_await wait_for_all_members(5s);
-        reduced_commit_interval.get("iceberg_catalog_commit_interval_ms")
-          .set_value(10000ms);
+        reduced_lag.get("iceberg_target_lag_ms").set_value(10000ms);
 
         co_await create_iceberg_topic(test_topic.tp);
         // create a kafka client for the cluster.
@@ -75,7 +74,7 @@ class PartitionTranslatorTestFixture
           });
     }
 
-    scoped_config reduced_commit_interval;
+    scoped_config reduced_lag;
 
     model::topic_namespace test_topic{
       model::kafka_namespace, model::topic{"test"}};
diff --git a/src/v/datalake/translation/BUILD b/src/v/datalake/translation/BUILD
index 47d974d2b86bd..5d6458b806366 100644
--- a/src/v/datalake/translation/BUILD
+++ b/src/v/datalake/translation/BUILD
@@ -145,6 +145,7 @@ redpanda_cc_library(
         "//src/v/ssx:semaphore",
         "//src/v/utils:prefix_logger",
         "//src/v/utils:retry_chain_node",
+        "//src/v/utils:to_string",
         "@seastar",
     ],
 )
diff --git a/src/v/datalake/translation/deps.cc b/src/v/datalake/translation/deps.cc
index b15d69542856a..2245667180eff 100644
--- a/src/v/datalake/translation/deps.cc
+++ b/src/v/datalake/translation/deps.cc
@@ -178,12 +178,11 @@ class partition_data_source : public data_source {
 
     ss::future<std::optional<kafka::offset>> wait_for_data_to_translate(
       std::optional<kafka::offset> last_translated_offset,
+      ss::lowres_clock::time_point deadline,
       ss::abort_source& as) final {
         // todo: add logic to wait for enough data to translate.
         // currently we just break even if a single batch of data is available
         constexpr auto poll_duration = 2s;
-        constexpr auto poll_limit = (poll_duration * 15) - 1s;
-        const auto deadline = ss::lowres_clock::now() + poll_limit;
         while (!has_more_data_to_translate(last_translated_offset)) {
             co_await ss::sleep_abortable(poll_duration, as);
             if (ss::lowres_clock::now() >= deadline) {
@@ -441,6 +440,20 @@ class partition_translation_context : public translation_context {
         }
     }
 
+    size_t flushed_bytes() const final {
+        size_t result = 0;
+        if (_in_progress_translation) {
+            result = _in_progress_translation->flushed_bytes();
+        }
+        return result;
+    }
+
+    std::optional<kafka::offset> last_translated_offset() const final {
+        return _in_progress_translation
+                 ? _in_progress_translation->last_translated_offset()
+                 : std::nullopt;
+    }
+
     ss::future<> flush() final {
         if (_in_progress_translation) {
             return _in_progress_translation->flush().then_wrapped(
@@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
+        return window_ended;
+    }
+
+private:
+    scheduling::clock::duration target_lag() const {
+        const auto& topic_cfg = _topics.get_topic_cfg(
+          model::topic_namespace_view{_partition->ntp()});
+        auto default_lag
+          = config::shard_local_cfg().iceberg_target_lag_ms.value();
+        if (!topic_cfg.has_value()) {
+            return std::chrono::duration_cast<scheduling::clock::duration>(
+              default_lag);
+        }
+        auto target_lag = topic_cfg->properties.iceberg_target_lag_ms.value_or(
+          default_lag);
+        return std::chrono::duration_cast<scheduling::clock::duration>(
+          target_lag);
+    }
+
+    void reset_to_next_window() { _current_window_end += target_lag(); }
+
+    ss::lw_shared_ptr<cluster::partition> _partition;
+    cluster::topic_table& _topics;
+    scheduling::clock::time_point _current_window_end;
+};
+
+std::unique_ptr<translation_lag_tracker>
+translation_lag_tracker::make_default_lag_tracker(
+  ss::lw_shared_ptr<cluster::partition> partition,
+  cluster::topic_table& topics) {
+    return std::make_unique<partition_lag_tracker>(
+      std::move(partition), topics);
+}
+
 } // namespace datalake::translation
diff --git a/src/v/datalake/translation/deps.h b/src/v/datalake/translation/deps.h
index 79cc91cee1426..b20b416dc86db 100644
--- a/src/v/datalake/translation/deps.h
+++ b/src/v/datalake/translation/deps.h
@@ -111,7 +111,9 @@ class data_source {
      * or nullopt if there is no data to translate.
      */
     virtual ss::future<std::optional<kafka::offset>> wait_for_data_to_translate(
-      std::optional<kafka::offset> last_translated_offset, ss::abort_source&)
+      std::optional<kafka::offset> last_translated_offset,
+      ss::lowres_clock::time_point deadline,
+      ss::abort_source&)
       = 0;
 
     virtual ss::future<std::optional<model::record_batch_reader>>
@@ -180,6 +182,17 @@ class translation_context {
      */
     virtual ss::future<> flush() = 0;
 
+    /**
+     * Returns the number of bytes that are flushed to the writer from the
+     * inflight translation.
+     */
+    virtual size_t flushed_bytes() const = 0;
+
+    /**
+     * Returns the last translated offset by the translator, if one exists.
+     */
+    virtual std::optional<kafka::offset> last_translated_offset() const = 0;
+
     /**
      * Reconciles the translator configurations.
      */
@@ -215,6 +228,22 @@ class translation_context {
       ss::lw_shared_ptr<translation_probe>);
 };
 
+class translation_lag_tracker {
+public:
+    translation_lag_tracker() = default;
+    translation_lag_tracker(const translation_lag_tracker&) = delete;
+    translation_lag_tracker& operator=(const translation_lag_tracker&) = delete;
+    translation_lag_tracker(translation_lag_tracker&&) = delete;
+    translation_lag_tracker& operator=(translation_lag_tracker&&) = delete;
+
+    virtual ~translation_lag_tracker() = default;
+
+    virtual bool should_finish_inflight_translation() = 0;
+
+    static std::unique_ptr<translation_lag_tracker> make_default_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition>, cluster::topic_table&);
+};
+
 std::unique_ptr<table_creator>
 make_default_table_creator(coordinator::frontend&);
 
diff --git a/src/v/datalake/translation/partition_translator.cc b/src/v/datalake/translation/partition_translator.cc
index 7d8dd1e70bc1f..2ebe0dfe081b2 100644
--- a/src/v/datalake/translation/partition_translator.cc
+++ b/src/v/datalake/translation/partition_translator.cc
@@ -12,6 +12,7 @@
 
 #include "datalake/logger.h"
 #include "resource_mgmt/io_priority.h"
+#include "utils/to_string.h"
 
 #include <seastar/coroutine/as_future.hh>
 #include <seastar/util/defer.hh>
@@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t partition_flushed_bytes_limit = 32_MiB;
+
 template<
   typename Func,
   typename ShouldRetry,
@@ -69,6 +78,7 @@ partition_translator::partition_translator(
   std::unique_ptr<coordinator_api> coordinator,
   std::unique_ptr<data_source> data_source,
   std::unique_ptr<translation_context> translation_ctx,
+  std::unique_ptr<translation_lag_tracker> lag_tracker,
   jitter_t jitter,
   std::chrono::milliseconds retry_max_timeout,
   std::chrono::milliseconds retry_initial_backoff)
@@ -76,6 +86,7 @@ partition_translator::partition_translator(
   , _coordinator(std::move(coordinator))
   , _data_source(std::move(data_source))
   , _translation_ctx(std::move(translation_ctx))
+  , _lag_tracking(std::move(lag_tracker))
   , _jitter{std::move(jitter)}
   , _retry_max_timeout(retry_max_timeout)
   , _retry_initial_backoff(retry_initial_backoff)
@@ -137,6 +148,7 @@ partition_translator::translate_when_notified(kafka::offset begin_offset) {
     if (!reader) {
         co_return;
     }
+    vlog(_logger.trace, "starting translation from offset: {}", begin_offset);
     ss::timer<scheduling::clock> cancellation_timer;
     cancellation_timer.set_callback([&as] { as.request_abort(); });
 
@@ -151,6 +163,18 @@ partition_translator::translate_when_notified(kafka::offset begin_offset) {
     });
 }
 
+bool partition_translator::should_finish_inflight_translation() const {
+    auto bytes_flushed_pending_upload = _translation_ctx->flushed_bytes();
+    auto lag_window_ended = _lag_tracking->should_finish_inflight_translation();
+    vlog(
+      _logger.trace,
+      "current bytes flushed: {}, lag window roll: {}",
+      bytes_flushed_pending_upload,
+      lag_window_ended);
+    return bytes_flushed_pending_upload >= partition_flushed_bytes_limit
+           || lag_window_ended;
+}
+
 ss::future<> partition_translator::translate_until_stopped() {
     const auto& id = _data_source->ntp();
     vassert(
@@ -188,7 +212,8 @@ ss::future<> partition_translator::translate_until_stopped() {
         if (reset_error) {
             vlog(
               _logger.warn,
-              "error updating highest translated offset: {}, translation will "
+              "error updating highest translated offset: {}, translation "
+              "will "
               "be retried",
               reset_error);
             continue;
@@ -197,65 +222,77 @@ ss::future<> partition_translator::translate_until_stopped() {
         // Update partition metrics. Note that last committed offset here
         // is NOT synchronized with outstanding commit operations. Therefore
         // if we reach this point before the most recent batch of files has
-        // been committed, the commit lag metric will be out of sync at least
-        // until 'wait_for_data' returns and we re-enter the loop.
+        // been committed, the commit lag metric will be out of sync at
+        // least until 'wait_for_data' returns and we re-enter the loop.
         _data_source->update_commit_lag(last_committed_offset);
         _data_source->update_translation_lag(last_translated_offset);
 
+        auto local_last_translated_offset = last_translated_offset;
+        if (
+          auto last_known_translated_offset
+          = _translation_ctx->last_translated_offset()) {
+            // A translation already in progress, start from where we left off.
+            local_last_translated_offset = last_known_translated_offset.value();
+        }
+
+        static constexpr auto data_wait_duration = 3s;
         // Wait until some data is ready to be translated.
         auto maybe_begin_offset
           = co_await _data_source->wait_for_data_to_translate(
-            last_translated_offset, _as);
+            local_last_translated_offset,
+            ss::lowres_clock::now() + data_wait_duration,
+            _as);
 
-        // if wait_for_data timed out (i.e. all trnaslatable records have
+        // if wait_for_data timed out (i.e. all translatable records have
         // been translated already), reenter the loop. this gives us an
         // opportunity to reconcile outstanding coordinator state, which is
         // helpful for keeping lag metrics up-to-date.
-        if (!maybe_begin_offset.has_value()) {
-            continue;
+        if (maybe_begin_offset) {
+            auto begin_offset = maybe_begin_offset.value();
+
+            // Notify the scheduler that there is some data to translate
+            _scheduler->notify_ready(id);
+
+            // wait for the scheduler to notify back that we've been given a
+            // time slice (i.e. scheduled in), then translate until the time
+            // slice expires or we run out of data
+            auto translate_f = co_await ss::coroutine::as_future<>(
+              translate_when_notified(begin_offset));
+
+            // inflight_translation_state tracks a single scheduled chunk of
+            // work, so we reset it to nullopt for the next time we're scheduled
+            // in
+            _inflight_translation_state.reset();
+
+            // Let the scheduler know we are done
+            _scheduler->notify_done(id);
+
+            if (translate_f.failed()) {
+                vlog(
+                  _logger.warn,
+                  "Translation attempt failed: {}, discarding state to reset "
+                  "translation",
+                  translate_f.get_exception());
+                co_await _translation_ctx->discard();
+                continue;
+            }
         }
 
-        auto begin_offset = maybe_begin_offset.value();
-
-        // Notify the scheduler that there is some data to translate
-        _scheduler->notify_ready(id);
-
-        // wait for the scheduler to notify back that we've been given a time
-        // slice (i.e. scheduled in), then translate until the time slice
-        // expires or we run out of data
-        auto translate_f = co_await ss::coroutine::as_future<>(
-          translate_when_notified(begin_offset));
-
-        // inflight_translation_state tracks a single scheduled chunk of work,
-        // so we reset it to nullopt for the next time we're scheduled in
-        _inflight_translation_state.reset();
-
-        // Let the scheduler know we are done
-        _scheduler->notify_done(id);
-
-        if (translate_f.failed()) {
-            vlog(
-              _logger.warn,
-              "Translation attempt failed: {}, discarding state to reset "
-              "translation",
-              translate_f.get_exception());
-            co_await _translation_ctx->discard();
+        if (!should_finish_inflight_translation()) {
+            scoped_set_jitter.cancel();
             continue;
         }
 
-        // TODO: add logic to skip finish on every iteration. This is
-        // predicated on whether we are close to the lag deadline or if too many
-        // bytes have accumulated
-        // With out this, we finish() on every translation which makes it
-        // equivalent to the logic without this patch.
-
         auto translation_result = co_await _translation_ctx->finish(rcn, _as);
         if (!translation_result) {
             vlog(_logger.warn, "Failed to translate, retrying");
             continue;
         }
 
-        if (begin_offset != translation_result->start_offset) {
+        // Check if the translated offset space is contiguous, if not make it
+        // so.
+        auto expected_begin = kafka::next_offset(last_translated_offset);
+        if (expected_begin != translation_result->start_offset) {
             // This is possible if there is a gap in offsets range, eg from
             // compaction. Normally that shouldn't be the case, as translation
             // enforces max_collectible_offset which prevents compaction or
@@ -268,9 +305,9 @@ ss::future<> partition_translator::translate_until_stopped() {
               _logger.info,
               "detected an offset range gap in [{}, {}), adjusting the begin "
               "offset to avoid gaps in coordinator tracked offsets.",
-              begin_offset,
+              expected_begin,
               translation_result->start_offset);
-            translation_result->start_offset = begin_offset;
+            translation_result->start_offset = expected_begin;
         }
 
         last_translated_offset = translation_result->last_offset;
@@ -311,8 +348,8 @@ ss::future<> partition_translator::init(
     _initialized = true;
     ssx::repeat_until_gate_closed_or_aborted(_gate, _as, [this] {
         return ss::with_scheduling_group(_sg, [this] {
-            return translate_until_stopped().handle_exception(
-              [this](const std::exception_ptr& ex) {
+            return translate_until_stopped()
+              .handle_exception([this](const std::exception_ptr& ex) {
                   auto log_level = ssx::is_shutdown_exception(ex)
                                      ? ss::log_level::debug
                                      : ss::log_level::warn;
@@ -322,6 +359,19 @@ ss::future<> partition_translator::init(
                     "[{}] Encountered exception in translation loop: {}",
                     ex,
                     _data_source->ntp());
+              })
+              .then([this] {
+                  // discard any inflight state and start from scratch
+                  return _translation_ctx->discard().then_wrapped(
+                    [this](ss::future<> f) {
+                        if (f.failed()) {
+                            vlog(
+                              _logger.warn,
+                              "Exception cleaning up inflight translation: {}",
+                              f.get_exception());
+                        }
+                        return ss::make_ready_future();
+                    });
               });
         });
     });
diff --git a/src/v/datalake/translation/partition_translator.h b/src/v/datalake/translation/partition_translator.h
index 72161859bd71b..52f19601fe38c 100644
--- a/src/v/datalake/translation/partition_translator.h
+++ b/src/v/datalake/translation/partition_translator.h
@@ -73,6 +73,7 @@ class partition_translator : public scheduling::translator {
       std::unique_ptr<coordinator_api>,
       std::unique_ptr<data_source>,
       std::unique_ptr<translation_context>,
+      std::unique_ptr<translation_lag_tracker>,
       jitter_t jitter,
       std::chrono::milliseconds retry_max_timeout,
       std::chrono::milliseconds retry_initial_backoff);
@@ -147,10 +148,18 @@ class partition_translator : public scheduling::translator {
     checkpoint_translation_result(
       retry_chain_node&, coordinator::translated_offset_range);
 
+    /**
+     * Returns true if the inflight translation should be flushed up on which
+     * all the files are rolled and uploaded to the cloud storage and the result
+     * will be checkpointed.
+     */
+    bool should_finish_inflight_translation() const;
+
     ss::scheduling_group _sg;
     std::unique_ptr<coordinator_api> _coordinator;
     std::unique_ptr<data_source> _data_source;
     std::unique_ptr<translation_context> _translation_ctx;
+    std::unique_ptr<translation_lag_tracker> _lag_tracking;
     // TODO: consider baking backoff into the scheduler on translation failure.
     jitter_t _jitter;
     std::chrono::milliseconds _retry_max_timeout;
diff --git a/src/v/datalake/translation/tests/partition_translator_tests.cc b/src/v/datalake/translation/tests/partition_translator_tests.cc
index 8b354b421c833..fefc2d9c44d62 100644
--- a/src/v/datalake/translation/tests/partition_translator_tests.cc
+++ b/src/v/datalake/translation/tests/partition_translator_tests.cc
@@ -79,6 +79,23 @@ class fake_test_ctx {
 
     void translation_attempt() { _num_translation_attempts++; }
 
+    void set_should_finish_inflight_translation(bool should_finish) {
+        _should_finish_inflight_translation = should_finish;
+    }
+    bool should_finish_inflight_translation() {
+        return _should_finish_inflight_translation;
+    }
+
+    const auto& translated_files() const { return _files; }
+
+    void add_translated_files(
+      const model::topic_partition& tp, chunked_vector<data_file> files) {
+        auto& partition_files = _files[tp];
+        for (auto& file : files) {
+            partition_files.push_back(std::move(file));
+        }
+    }
+
     ss::future<> wait_for_translation_attempts(
       size_t attempts, std::chrono::seconds timeout = 10s) {
         auto target = _num_translation_attempts + attempts;
@@ -107,6 +124,10 @@ class fake_test_ctx {
     bool _error_on_translation{false};
     bool _error_on_flush{false};
     size_t _num_translation_attempts{0};
+    bool _should_finish_inflight_translation{true};
+
+    absl::flat_hash_map<model::topic_partition, chunked_vector<data_file>>
+      _files;
 };
 
 class fake_data_src : public data_source {
@@ -124,9 +145,15 @@ class fake_data_src : public data_source {
 
     ss::future<std::optional<kafka::offset>> wait_for_data_to_translate(
       std::optional<kafka::offset> last_translated_offset,
+      ss::lowres_clock::time_point,
       ss::abort_source& as) final {
-        while (!as.abort_requested()) {
+        auto deadline = ss::lowres_clock::now() + 5ms;
+        while (!as.abort_requested() && ss::lowres_clock::now() < deadline) {
             auto offset = max_offset_for_translation();
+            vlog(
+              test_logger.trace,
+              "current max offset: {}",
+              offset.value_or(kafka::offset{}));
             if (offset && offset.value() >= kafka::offset{0} && (!last_translated_offset || offset.value() > last_translated_offset.value())) {
                 co_return last_translated_offset
                   ? kafka::next_offset(last_translated_offset.value())
@@ -140,7 +167,7 @@ class fake_data_src : public data_source {
     ss::future<std::optional<model::record_batch_reader>> make_log_reader(
       kafka::offset o, ss::io_priority_class, ss::abort_source&) final {
         auto batches = co_await model::test::make_random_batches(
-          kafka::offset_cast(o), 5, true);
+          kafka::offset_cast(o), 500, false);
         auto reader = model::make_generating_record_batch_reader(
           [batches = std::move(batches)]() mutable {
               return ss::make_ready_future<model::record_batch_reader::data_t>(
@@ -199,6 +226,10 @@ class fake_coordinator_api : public coordinator_api {
         auto it = _last_added_offsets.find(request.tp);
         if (it == _last_added_offsets.end() || it->second < last.last_offset) {
             _last_added_offsets[request.tp] = last.last_offset;
+            for (auto& range : request.ranges) {
+                _test_ctx.add_translated_files(
+                  request.tp, std::move(range.files));
+            }
         } else {
             reply.errc = errc::failed;
         }
@@ -220,6 +251,7 @@ class fake_coordinator_api : public coordinator_api {
 private:
     absl::flat_hash_map<model::topic_partition, kafka::offset>
       _last_added_offsets;
+
     fake_test_ctx& _test_ctx;
 };
 
@@ -248,7 +280,9 @@ class fake_translation_ctx : public translation_context {
         }
         auto min = model::offset_cast(batches.begin()->base_offset());
         auto max = model::offset_cast(batches.back().last_offset());
-
+        for (auto& batch : batches) {
+            _translated_bytes += batch.size_bytes();
+        }
         // keep track of translation boundaries for result propagation
         _min_offset_translated = std::min(
           min, _min_offset_translated.value_or(kafka::offset::max()));
@@ -256,10 +290,19 @@ class fake_translation_ctx : public translation_context {
           max, _max_offset_translated.value_or(kafka::offset::min()));
     }
 
+    std::optional<kafka::offset> last_translated_offset() const final {
+        return _max_offset_translated;
+    }
+
+    size_t flushed_bytes() const final { return _flushed_bytes; }
+
     ss::future<> flush() final {
         if (_test_ctx.error_on_flush()) {
-            return ss::make_exception_future("flush error simulation");
+            return ss::make_exception_future(
+              std::runtime_error("flush error simulation"));
         }
+        _flushed_bytes += _translated_bytes;
+        _translated_bytes = 0;
         return ss::make_ready_future();
     }
 
@@ -273,22 +316,46 @@ class fake_translation_ctx : public translation_context {
             result.start_offset = _min_offset_translated.value();
             result.last_offset = _max_offset_translated.value();
         }
+        data_file file;
+        file.file_size_bytes = _flushed_bytes;
+        result.files.push_back(std::move(file));
+        _flushed_bytes = 0;
+        _min_offset_translated.reset();
+        _max_offset_translated.reset();
         return ss::make_ready_future<std::optional<translated_offset_range>>(
           std::move(result));
     }
 
     ss::future<> discard() final {
         _inflight_translation = false;
+        _flushed_bytes = 0;
+        _min_offset_translated.reset();
+        _max_offset_translated.reset();
         return ss::make_ready_future();
     }
 
 private:
+    size_t _translated_bytes{0};
+    size_t _flushed_bytes{0};
     std::optional<kafka::offset> _min_offset_translated;
     std::optional<kafka::offset> _max_offset_translated;
     fake_test_ctx& _test_ctx;
     bool _inflight_translation{false};
 };
 
+class fake_lag_tracker : public translation_lag_tracker {
+public:
+    explicit fake_lag_tracker(fake_test_ctx& test_ctx)
+      : _test_ctx(test_ctx) {}
+
+    bool should_finish_inflight_translation() final {
+        return _test_ctx.should_finish_inflight_translation();
+    }
+
+private:
+    fake_test_ctx& _test_ctx;
+};
+
 class partition_translator_fixture : public scheduling::scheduler_fixture {
 public:
     ss::future<> SetUpAsync() override {
@@ -309,6 +376,7 @@ class partition_translator_fixture : public scheduling::scheduler_fixture {
             std::make_unique<fake_coordinator_api>(ctx),
             std::make_unique<fake_data_src>(ctx),
             std::make_unique<fake_translation_ctx>(ctx),
+            std::make_unique<fake_lag_tracker>(ctx),
             simple_time_jitter<ss::lowres_clock, std::chrono::milliseconds>{
               retry_max_timeout, retry_initial_backoff},
             retry_max_timeout,
@@ -332,12 +400,12 @@ TEST_F_CORO(partition_translator_fixture, test_cleanup_on_translation_error) {
     auto& test_ctx = make_test_context();
     test_ctx.set_error_on_translation(true);
     co_await add_translator(test_ctx);
-    co_await test_ctx.wait_for_translation_attempts(30);
+    co_await test_ctx.wait_for_translation_attempts(5);
     // Ensure translation does not make any progress
     ASSERT_LT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
     // undo
     test_ctx.set_error_on_translation(false);
-    co_await test_ctx.wait_for_translation_attempts(30);
+    co_await test_ctx.wait_for_translation_attempts(5);
     ASSERT_GT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
 }
 
@@ -345,12 +413,12 @@ TEST_F_CORO(partition_translator_fixture, test_cleanup_on_flush_error) {
     auto& test_ctx = make_test_context();
     test_ctx.set_error_on_flush(true);
     co_await add_translator(test_ctx);
-    co_await test_ctx.wait_for_translation_attempts(30);
+    co_await test_ctx.wait_for_translation_attempts(5);
     // Ensure translation does not make any progress
     ASSERT_LT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
     // undo
     test_ctx.set_error_on_flush(false);
-    co_await test_ctx.wait_for_translation_attempts(30);
+    co_await test_ctx.wait_for_translation_attempts(5);
     ASSERT_GT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
 }
 
@@ -359,7 +427,6 @@ TEST_F_CORO(partition_translator_fixture, test_coordinator_retries) {
     // simulate no leader conditions
     test_ctx.set_error_on_add_translated_files(true);
     co_await add_translator(test_ctx);
-    co_await test_ctx.wait_for_translation_attempts(10);
     // Ensure translation does not make any progress
     ASSERT_LT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
     test_ctx.set_error_on_add_translated_files(false);
@@ -367,4 +434,41 @@ TEST_F_CORO(partition_translator_fixture, test_coordinator_retries) {
     ASSERT_GT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
 }
 
-// todo: add a test for batching once the functionality is checked in
+TEST_F_CORO(partition_translator_fixture, test_batching) {
+    auto& test_ctx = make_test_context();
+    // Let the translator batch to the limit
+    test_ctx.set_should_finish_inflight_translation(false);
+    co_await add_translator(test_ctx);
+
+    auto stop = false;
+    auto produce_f = ss::do_until(
+      [&stop] { return stop; },
+      [&test_ctx]() {
+          auto current = test_ctx.max_translatable_offset();
+          test_ctx.set_max_translatable_offset(
+            current + kafka::offset_delta(10000));
+          return ss::sleep(1ms);
+      });
+
+    std::exception_ptr ex = nullptr;
+    try {
+        RPTEST_REQUIRE_EVENTUALLY_CORO(10s, [&test_ctx]() {
+            const auto& files = test_ctx.translated_files();
+            const auto it = files.find(test_ctx.ntp().tp);
+            if (it != files.end()) {
+                vlog(test_logger.trace, "translated files: {}", it->second);
+            }
+            return it != files.end() && it->second.size() >= 2
+                   && std::ranges::all_of(it->second, [](const auto& entry) {
+                          return entry.file_size_bytes >= 32_MiB;
+                      });
+        });
+    } catch (...) {
+        ex = std::current_exception();
+    }
+    stop = true;
+    co_await std::move(produce_f);
+    if (ex) {
+        RPTEST_FAIL_CORO(fmt::to_string(ex));
+    }
+}
diff --git a/src/v/datalake/translation_task.cc b/src/v/datalake/translation_task.cc
index d6d337cf175d4..ae1c76747824c 100644
--- a/src/v/datalake/translation_task.cc
+++ b/src/v/datalake/translation_task.cc
@@ -209,6 +209,10 @@ ss::future<> translation_task::translate_once(
       std::move(reader), _read_timeout + model::timeout_clock::now(), as);
 }
 
+size_t translation_task::flushed_bytes() const {
+    return _multiplexer.flushed_bytes();
+}
+
 ss::future<checked<void, translation_task::errc>> translation_task::flush() {
     auto result = co_await _multiplexer.flush_writers();
     if (result != writer_error::ok) {
@@ -218,6 +222,10 @@ ss::future<checked<void, translation_task::errc>> translation_task::flush() {
     co_return outcome::success();
 }
 
+std::optional<kafka::offset> translation_task::last_translated_offset() const {
+    return _multiplexer.last_translated_offset();
+}
+
 ss::future<
   checked<coordinator::translated_offset_range, translation_task::errc>>
 translation_task::finish(
diff --git a/src/v/datalake/translation_task.h b/src/v/datalake/translation_task.h
index b9e5f1dd65037..15e395cd4e3dd 100644
--- a/src/v/datalake/translation_task.h
+++ b/src/v/datalake/translation_task.h
@@ -57,12 +57,22 @@ class translation_task {
     ss::future<>
     translate_once(model::record_batch_reader reader, ss::abort_source&);
 
+    /**
+     * Current number of bytes flushed by all the open writers.
+     */
+    size_t flushed_bytes() const;
+
     /**
      * Flushes all the inflight translated bytes and releases memory
      * reservations. May not be called while translation is in progress.
      */
     ss::future<checked<void, errc>> flush();
 
+    /**
+     * Returns the last translated offset, if one exists
+     */
+    std::optional<kafka::offset> last_translated_offset() const;
+
     /**
      * Uploads the resulting translated files to the object store. The tasks
      * accepts an abort source indicating when the upload retires should be
diff --git a/tests/rptest/tests/datalake/batching_test.py b/tests/rptest/tests/datalake/batching_test.py
new file mode 100644
index 0000000000000..2f53ad6678d41
--- /dev/null
+++ b/tests/rptest/tests/datalake/batching_test.py
@@ -0,0 +1,95 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Licensed as a Redpanda Enterprise file under the Redpanda Community
+# License (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# https://github.com/redpanda-data/redpanda/blob/master/licenses/rcl.md
+
+from rptest.services.catalog_service import CatalogType
+from rptest.services.cluster import cluster
+from rptest.services.kgo_verifier_services import KgoVerifierProducer
+from rptest.services.redpanda import SISettings
+from rptest.tests.datalake.datalake_services import DatalakeServices
+from rptest.tests.datalake.query_engine_base import QueryEngineType
+from rptest.tests.datalake.utils import supported_storage_types
+from rptest.tests.redpanda_test import RedpandaTest
+from rptest.utils.mode_checks import skip_debug_mode
+from ducktape.utils.util import wait_until
+
+from ducktape.mark import matrix
+
+
+class DatalakeBatchingTest(RedpandaTest):
+    def __init__(self, test_ctx, *args, **kwargs):
+        super(DatalakeBatchingTest,
+              self).__init__(test_ctx,
+                             num_brokers=1,
+                             si_settings=SISettings(test_context=test_ctx),
+                             extra_rp_conf={
+                                 "iceberg_enabled": True,
+                                 "iceberg_catalog_commit_interval_ms": 100
+                             },
+                             *args,
+                             **kwargs)
+        self.test_ctx = test_ctx
+        self.topic_name = "test"
+
+    def setUp(self):
+        # redpanda will be started by DatalakeServices
+        pass
+
+    @cluster(num_nodes=4)
+    @skip_debug_mode
+    @matrix(cloud_storage_type=supported_storage_types(),
+            query_engine=[QueryEngineType.SPARK],
+            catalog_type=[CatalogType.REST_JDBC],
+            expect_large_files=[True, False])
+    def test_batching(self, cloud_storage_type, query_engine, catalog_type,
+                      expect_large_files):
+        """Test ensures that the broker produces sufficiently large parquet files on topics with large target lag."""
+
+        with DatalakeServices(self.test_ctx,
+                              redpanda=self.redpanda,
+                              include_query_engines=[query_engine],
+                              catalog_type=catalog_type) as dl:
+            lag = 60 * 60 * 1000 if expect_large_files else 60 * 1000
+            rate = 30 * 2**20 if expect_large_files else 2**20
+            dl.create_iceberg_enabled_topic(self.topic_name,
+                                            partitions=1,
+                                            target_lag_ms=lag)
+
+            producer = KgoVerifierProducer(self.test_ctx,
+                                           self.redpanda,
+                                           self.topic_name,
+                                           msg_size=4096,
+                                           msg_count=2**30,
+                                           rate_limit_bps=rate)
+            producer.start()
+
+            table_name = f"redpanda.{self.topic_name}"
+            spark = dl.spark()
+
+            min_file_count = 2
+            batch_file_size = 32 * 1024 * 1024
+
+            def wait_for_large_files():
+                try:
+                    file_count = spark.run_query_fetch_one(
+                        f"select count(*) from {table_name}.files")[0]
+                    file_size = spark.run_query_fetch_one(
+                        f"select min(file_size_in_bytes) from {table_name}.files"
+                    )[0]
+                    self.redpanda.logger.debug(
+                        f"count: {file_count}, size: {file_size}")
+                    file_size_checks_out = file_size >= batch_file_size if expect_large_files else file_size < batch_file_size
+                    return file_count >= min_file_count and file_size_checks_out
+                except:
+                    return False
+
+            wait_until(
+                wait_for_large_files,
+                timeout_sec=180,
+                backoff_sec=10,
+                err_msg=
+                "Timed out waiting for batched parquet files to be created")
diff --git a/tests/rptest/tests/datalake/datalake_e2e_test.py b/tests/rptest/tests/datalake/datalake_e2e_test.py
index 3ec31e210ff85..d804f920b06cb 100644
--- a/tests/rptest/tests/datalake/datalake_e2e_test.py
+++ b/tests/rptest/tests/datalake/datalake_e2e_test.py
@@ -383,7 +383,8 @@ def test_lag_metrics(self, cloud_storage_type):
 
             dl.create_iceberg_enabled_topic(self.topic_name,
                                             partitions=1,
-                                            replicas=3)
+                                            replicas=3,
+                                            target_lag_ms=10000)
             topic_leader = self.redpanda.partitions(self.topic_name)[0].leader
             count = randint(12, 21)
             dl.produce_to_topic(self.topic_name, 1, msg_count=count)
@@ -433,7 +434,8 @@ def test_rest_catalog_metrics(self, cloud_storage_type):
 
             dl.create_iceberg_enabled_topic(self.topic_name,
                                             partitions=1,
-                                            replicas=3)
+                                            replicas=3,
+                                            target_lag_ms=10000)
             dl.produce_to_topic(self.topic_name, 1024, 10)
 
             # Wait until we have committed to the table -- this implies several
@@ -476,7 +478,8 @@ def test_rest_catalog_metrics(self, cloud_storage_type):
 
             dl.create_iceberg_enabled_topic(self.topic_name,
                                             partitions=1,
-                                            replicas=3)
+                                            replicas=3,
+                                            target_lag_ms=10000)
             dl.produce_to_topic(self.topic_name, 1024, 10)
             wait_until(lambda: self.redpanda.metric_sum(
                 timeouts_metric, MetricsEndpoint.PUBLIC_METRICS) > 0,
diff --git a/tests/rptest/tests/datalake/datalake_services.py b/tests/rptest/tests/datalake/datalake_services.py
index d85361939a479..ab0cb55653d40 100644
--- a/tests/rptest/tests/datalake/datalake_services.py
+++ b/tests/rptest/tests/datalake/datalake_services.py
@@ -163,8 +163,10 @@ def create_iceberg_enabled_topic(self,
                                      partitions=1,
                                      replicas=1,
                                      iceberg_mode="key_value",
+                                     target_lag_ms=10000,
                                      config: dict[str, Any] = dict()):
         config[TopicSpec.PROPERTY_ICEBERG_MODE] = iceberg_mode
+        config[TopicSpec.PROPERTY_ICEBERG_TARGET_LAG_MS] = target_lag_ms
         rpk = RpkTool(self.redpanda)
         rpk.create_topic(topic=name,
                          partitions=partitions,
@@ -239,7 +241,7 @@ def translation_done():
     def wait_for_translation(self,
                              topic,
                              msg_count,
-                             timeout=30,
+                             timeout=60,
                              backoff_sec=5,
                              table_override=None):
         table_name = topic
diff --git a/tests/rptest/tests/datalake/recovery_mode_test.py b/tests/rptest/tests/datalake/recovery_mode_test.py
index 433eefce3ecd2..7ea9212904edf 100644
--- a/tests/rptest/tests/datalake/recovery_mode_test.py
+++ b/tests/rptest/tests/datalake/recovery_mode_test.py
@@ -86,8 +86,8 @@ def test_recovery_mode(self, cloud_storage_type, catalog_type):
             dl.produce_to_topic("foo", 1024, count)
             dl.produce_to_topic("bar", 1024, count)
 
-            dl.wait_for_translation("foo", msg_count=2 * count)
-            dl.wait_for_translation("bar", msg_count=count)
+            dl.wait_for_translation("foo", msg_count=2 * count, timeout=120)
+            dl.wait_for_translation("bar", msg_count=count, timeout=120)
 
     @cluster(num_nodes=6)
     @matrix(cloud_storage_type=supported_storage_types(),
@@ -121,5 +121,5 @@ def test_disabled_partitions(self, cloud_storage_type, catalog_type):
             dl.produce_to_topic("foo", 1024, count)
             dl.produce_to_topic("bar", 1024, count)
 
-            dl.wait_for_translation("foo", msg_count=2 * count)
-            dl.wait_for_translation("bar", msg_count=count)
+            dl.wait_for_translation("foo", msg_count=2 * count, timeout=120)
+            dl.wait_for_translation("bar", msg_count=count, timeout=120)


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR aims to add batching support to the translator by updating the datalake tests and service configuration parameters. Key changes include:
- Adding a new parameter (target_lag_ms) to topic creation calls in the end-to-end tests.
- Updating the default parameter in create_iceberg_enabled_topic and aligning it with the new testing values.
- Increasing timeout values in translation-related functions to support the batched translator behavior.

### Reviewed Changes

| File | Description |
| ---- | ----------- |
| tests/rptest/tests/datalake/datalake_e2e_test.py | Added target_lag_ms parameter to topic creation in relevant tests |
| tests/rptest/tests/datalake/datalake_services.py | Introduced target_lag_ms default in topic creation and increased timeouts in translation functions |

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

<details>
<summary>Comments suppressed due to low confidence (4)</summary>

**tests/rptest/tests/datalake/datalake_e2e_test.py:386**
* [nitpick] The test sets target_lag_ms to 10000, which differs from the services default of 60000. Verify that this discrepancy is intentional and that it aligns with the batching behavior requirements.
```
replicas=3, target_lag_ms=10000
```
**tests/rptest/tests/datalake/datalake_services.py:166**
* [nitpick] The default value for target_lag_ms is set to 60000 while tests override it with 10000. Consider reviewing these values for consistency unless they serve distinct purposes.
```
target_lag_ms=60000,
```
**tests/rptest/tests/datalake/datalake_services.py:216**
* [nitpick] The timeout in wait_for_translation_until_offset was increased from 30 to 120 seconds. Confirm that this extension is necessary for batching support and document the rationale if needed.
```
timeout=120,
```
**tests/rptest/tests/datalake/datalake_services.py:244**
* [nitpick] The timeout in wait_for_translation was increased from 30 to 120 seconds. Ensure that this change is appropriate for the new batching implementation and add inline documentation if required.
```
timeout=120,
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T20:26:08Z
user: login: oleiman
body: looks good to me overall. few general questions

nitpick:

>Files are not rolled once they hit 32 MiB

"Files **are** rolled once they hit 32 MiB", right?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T21:09:23Z
user: login: bharathv
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T21:50:55Z
user: login: oleiman
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T21:55:35Z
user: login: oleiman
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T21:57:51Z
user: login: andrwng
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T22:00:13Z
user: login: bharathv
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T05:19:38Z
user: login: andrwng
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T07:31:06Z
user: login: andrwng
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T07:34:09Z
user: login: andrwng
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T07:34:57Z
user: login: mmaslankaprv
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T07:57:12Z
user: login: bharathv
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T15:35:37Z

Review comments:
diff_hunk: @@ -322,6 +323,19 @@ ss::future<> partition_translator::init(
                     "[{}] Encountered exception in translation loop: {}",
                     ex,
                     _data_source->ntp());
+              })
+              .then([this] {
+                  // discard any inflight state and start from scratch
+                  return _translation_ctx->discard().then_wrapped(
+                    [this](ss::future<> f) {
+                        if (f.failed()) {
+                            vlog(
+                              _logger.warn,
+                              "Exception cleaning up inflight translation: {}",
+                              f.get_exception());
path: src/v/datalake/translation/partition_translator.cc
user: login: oleiman
body: is logging all we can do if discard fails? is it actually safe to resume right away? I think a code comment would help here (e.g. if this is a "best effort" discard or something).

created_at: 2025-03-11T16:12:02Z
author_association: MEMBER
diff_hunk: @@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t cumulative_flushed_byte_limit = 32_MiB;
path: src/v/datalake/translation/partition_translator.cc
user: login: oleiman
body: I think the node-wide total "uncommitted flushed bytes" is effectively unbounded, right. iirc @dotnwat was asking about this the other day...is it still a concern? or is the main thing that we provide some interface so that space management can see the running total?
created_at: 2025-03-11T20:22:54Z
author_association: MEMBER
diff_hunk: @@ -198,65 +222,77 @@ ss::future<> partition_translator::translate_until_stopped() {
         // Update partition metrics. Note that last committed offset here
         // is NOT synchronized with outstanding commit operations. Therefore
         // if we reach this point before the most recent batch of files has
-        // been committed, the commit lag metric will be out of sync at least
-        // until 'wait_for_data' returns and we re-enter the loop.
+        // been committed, the commit lag metric will be out of sync at
+        // least until 'wait_for_data' returns and we re-enter the loop.
         _data_source->update_commit_lag(last_committed_offset);
         _data_source->update_translation_lag(last_translated_offset);
 
+        auto local_last_translated_offset = last_translated_offset;
+        if (
+          auto last_known_translated_offset
+          = _translation_ctx->last_translated_offset()) {
+            // A translation already in progress, start from where we left off.
+            local_last_translated_offset = last_known_translated_offset.value();
+        }
+
+        static constexpr auto data_wait_duration = 3s;
path: src/v/datalake/translation/partition_translator.cc
user: login: oleiman
body: q: how did you pick 3s? upper bound for how "late" we finish inflight translation? maybe a code comment would help.
created_at: 2025-03-11T20:25:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,83 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Licensed as a Redpanda Enterprise file under the Redpanda Community
+# License (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# https://github.com/redpanda-data/redpanda/blob/master/licenses/rcl.md
+
+from rptest.services.catalog_service import CatalogType
+from rptest.services.cluster import cluster
+from rptest.services.kgo_verifier_services import KgoVerifierProducer
+from rptest.services.redpanda import SISettings
+from rptest.tests.datalake.datalake_services import DatalakeServices
+from rptest.tests.datalake.query_engine_base import QueryEngineType
+from rptest.tests.datalake.utils import supported_storage_types
+from rptest.tests.redpanda_test import RedpandaTest
+from rptest.utils.mode_checks import skip_debug_mode
+from ducktape.utils.util import wait_until
+
+from ducktape.mark import matrix
+
+
+class DatalakeBatchingTest(RedpandaTest):
+    def __init__(self, test_ctx, *args, **kwargs):
+        super(DatalakeBatchingTest,
+              self).__init__(test_ctx,
+                             num_brokers=1,
+                             si_settings=SISettings(test_context=test_ctx),
+                             extra_rp_conf={"iceberg_enabled": True},
+                             *args,
+                             **kwargs)
+        self.test_ctx = test_ctx
+        self.topic_name = "test"
+
+    def setUp(self):
+        # redpanda will be started by DatalakeServices
+        pass
+
+    @cluster(num_nodes=4)
+    @skip_debug_mode
+    @matrix(cloud_storage_type=supported_storage_types(),
+            query_engine=[QueryEngineType.SPARK],
+            catalog_type=[CatalogType.REST_JDBC])
+    def test_batching(self, cloud_storage_type, query_engine, catalog_type):
path: tests/rptest/tests/datalake/batching_test.py
user: login: oleiman
body: Even with the dummy tracker, maybe we could have a test that e.g. a low tput topic still gets flushed _eventually_?
created_at: 2025-03-11T21:08:51Z
author_association: MEMBER
diff_hunk: @@ -322,6 +323,19 @@ ss::future<> partition_translator::init(
                     "[{}] Encountered exception in translation loop: {}",
                     ex,
                     _data_source->ntp());
+              })
+              .then([this] {
+                  // discard any inflight state and start from scratch
+                  return _translation_ctx->discard().then_wrapped(
+                    [this](ss::future<> f) {
+                        if (f.failed()) {
+                            vlog(
+                              _logger.warn,
+                              "Exception cleaning up inflight translation: {}",
+                              f.get_exception());
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: ya discard() should technically reset things, thats the intention. I'll add a comment.
created_at: 2025-03-11T21:37:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
+        return window_ended;
+    }
+
+private:
+    scheduling::clock::duration target_lag() const {
+        const auto& topic_cfg = _topics.get_topic_cfg(
+          model::topic_namespace_view{_partition->ntp()});
+        auto default_lag
+          = config::shard_local_cfg().iceberg_target_lag_ms.value();
+        if (!topic_cfg.has_value()) {
+            return std::chrono::duration_cast<scheduling::clock::duration>(
+              default_lag);
+        }
+        auto target_lag = topic_cfg->properties.iceberg_target_lag_ms.value_or(
+          default_lag);
+        return std::chrono::duration_cast<scheduling::clock::duration>(
+          target_lag);
+    }
+
+    void reset_to_next_window() { _current_window_end += target_lag(); }
path: src/v/datalake/translation/deps.cc
user: login: andrwng
body: Curious why not now + target_lag()? Though i guess it doesn't matter if this is all throwaway
created_at: 2025-03-11T21:38:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -180,6 +182,17 @@ class translation_context {
      */
     virtual ss::future<> flush() = 0;
 
+    /**
+     * Returns the number of bytes that are flushed to the writer from the
+     * inflight translation.
+     */
+    virtual size_t flushed_bytes() const = 0;
+
+    /**
+     * Returns the last translated offset by the translator, if one exists.
+     */
+    virtual std::optional<kafka::offset> last_translated_offset() const = 0;
path: src/v/datalake/translation/deps.h
user: login: andrwng
body: Meant for first commit?
created_at: 2025-03-11T21:39:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
path: src/v/datalake/translation/deps.cc
user: login: andrwng
body: Just noting that at first glance bit surprising for this to be resetting the window too. I would've expected callers to need to reset this explicitly, e.g. when successfully checkpointing translation success or something
created_at: 2025-03-11T21:42:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t cumulative_flushed_byte_limit = 32_MiB;
path: src/v/datalake/translation/partition_translator.cc
user: login: andrwng
body: >this is _per_ (partition) translator.

nit: maybe this should be `partition_flushed_bytes_limit`?
created_at: 2025-03-11T21:47:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t cumulative_flushed_byte_limit = 32_MiB;
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: > I think the node-wide total "uncommitted flushed bytes" is effectively unbounded

I wouldn't say it is unbounded. This limit along with a small default lag window of 1min puts a reasonable bound per partition. That said even if scheduler were to use significant disk usage, that will force space managment to aggressively trim segment data (not ideal but the system should hold up, see comment below).

> is the main thing that we provide some interface so that space management can see the running total?

This is already there. Check datalake_manager::disk_usage()
The conclusion was to force scheduler to run under a disk budget which is a fast follow after GA.
created_at: 2025-03-11T21:48:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -198,65 +222,77 @@ ss::future<> partition_translator::translate_until_stopped() {
         // Update partition metrics. Note that last committed offset here
         // is NOT synchronized with outstanding commit operations. Therefore
         // if we reach this point before the most recent batch of files has
-        // been committed, the commit lag metric will be out of sync at least
-        // until 'wait_for_data' returns and we re-enter the loop.
+        // been committed, the commit lag metric will be out of sync at
+        // least until 'wait_for_data' returns and we re-enter the loop.
         _data_source->update_commit_lag(last_committed_offset);
         _data_source->update_translation_lag(last_translated_offset);
 
+        auto local_last_translated_offset = last_translated_offset;
+        if (
+          auto last_known_translated_offset
+          = _translation_ctx->last_translated_offset()) {
+            // A translation already in progress, start from where we left off.
+            local_last_translated_offset = last_known_translated_offset.value();
+        }
+
+        static constexpr auto data_wait_duration = 3s;
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: actually random, may be its ok to wait for more, I also plan to do a follow up for better row group sizes (wait more to accumulate more per flush).
created_at: 2025-03-11T21:49:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,83 @@
+# Copyright 2025 Redpanda Data, Inc.
+#
+# Licensed as a Redpanda Enterprise file under the Redpanda Community
+# License (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# https://github.com/redpanda-data/redpanda/blob/master/licenses/rcl.md
+
+from rptest.services.catalog_service import CatalogType
+from rptest.services.cluster import cluster
+from rptest.services.kgo_verifier_services import KgoVerifierProducer
+from rptest.services.redpanda import SISettings
+from rptest.tests.datalake.datalake_services import DatalakeServices
+from rptest.tests.datalake.query_engine_base import QueryEngineType
+from rptest.tests.datalake.utils import supported_storage_types
+from rptest.tests.redpanda_test import RedpandaTest
+from rptest.utils.mode_checks import skip_debug_mode
+from ducktape.utils.util import wait_until
+
+from ducktape.mark import matrix
+
+
+class DatalakeBatchingTest(RedpandaTest):
+    def __init__(self, test_ctx, *args, **kwargs):
+        super(DatalakeBatchingTest,
+              self).__init__(test_ctx,
+                             num_brokers=1,
+                             si_settings=SISettings(test_context=test_ctx),
+                             extra_rp_conf={"iceberg_enabled": True},
+                             *args,
+                             **kwargs)
+        self.test_ctx = test_ctx
+        self.topic_name = "test"
+
+    def setUp(self):
+        # redpanda will be started by DatalakeServices
+        pass
+
+    @cluster(num_nodes=4)
+    @skip_debug_mode
+    @matrix(cloud_storage_type=supported_storage_types(),
+            query_engine=[QueryEngineType.SPARK],
+            catalog_type=[CatalogType.REST_JDBC])
+    def test_batching(self, cloud_storage_type, query_engine, catalog_type):
path: tests/rptest/tests/datalake/batching_test.py
user: login: bharathv
body: good idea, will parameterize this test for different throughputs.
created_at: 2025-03-11T21:50:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -359,12 +427,48 @@ TEST_F_CORO(partition_translator_fixture, test_coordinator_retries) {
     // simulate no leader conditions
     test_ctx.set_error_on_add_translated_files(true);
     co_await add_translator(test_ctx);
-    co_await test_ctx.wait_for_translation_attempts(10);
     // Ensure translation does not make any progress
     ASSERT_LT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
     test_ctx.set_error_on_add_translated_files(false);
     co_await test_ctx.wait_for_translation_attempts(10);
     ASSERT_GT_CORO(test_ctx.max_translated_offset(), kafka::offset{0});
 }
 
-// todo: add a test for batching once the functionality is checked in
+TEST_F_CORO(partition_translator_fixture, test_batching) {
+    auto& test_ctx = make_test_context();
+    // Let the translator batch to the limit
+    test_ctx.set_should_finish_inflight_translation(false);
+    co_await add_translator(test_ctx);
+
+    auto stop = false;
+    auto produce_f = ss::do_until(
+      [&stop] { return stop; },
+      [&test_ctx]() {
+          auto current = test_ctx.max_translatable_offset();
+          test_ctx.set_max_translatable_offset(
+            current + kafka::offset_delta(10000));
+          return ss::sleep(1ms);
+      });
+
+    std::exception_ptr ex = nullptr;
+    try {
+        RPTEST_REQUIRE_EVENTUALLY_CORO(10s, [&test_ctx]() {
+            const auto& files = test_ctx.translated_files();
+            const auto it = files.find(test_ctx.ntp().tp);
+            if (it != files.end()) {
+                vlog(test_logger.trace, "translated files: {}", it->second);
+            }
+            return it != files.end() && it->second.size() >= 2
+                   && std::ranges::all_of(it->second, [](const auto& entry) {
+                          return entry.file_size_bytes >= 32_MiB;
+                      });
+        });
+    } catch (...) {
+        ex = std::current_exception();
+    }
+    stop = true;
+    co_await std::move(produce_f);
+    if (ex) {
+        std::rethrow_exception(ex);
path: src/v/datalake/translation/tests/partition_translator_tests.cc
user: login: andrwng
body: nit: RPTEST_FAIL_CORO(fmt::to_string(ex))`? Not sure about gtest but i know debugging exceptions thrown in test bodies with boost test is really annoying if you aren't the author of the test (and maybe still even if you are!)
created_at: 2025-03-11T21:55:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t cumulative_flushed_byte_limit = 32_MiB;
path: src/v/datalake/translation/partition_translator.cc
user: login: oleiman
body: yeah sorry, i mean "unbounded" in the sense that we could have arbitrarily many partitions with inflight translated data sitting on disk (even with an upper bound per partition). 

> conclusion was to force scheduler to run under a disk budget... after GA

I think that answers my question though 👍 
created_at: 2025-03-11T21:55:34Z
author_association: MEMBER
diff_hunk: @@ -322,6 +323,19 @@ ss::future<> partition_translator::init(
                     "[{}] Encountered exception in translation loop: {}",
                     ex,
                     _data_source->ntp());
+              })
+              .then([this] {
+                  // discard any inflight state and start from scratch
+                  return _translation_ctx->discard().then_wrapped(
+                    [this](ss::future<> f) {
+                        if (f.failed()) {
+                            vlog(
+                              _logger.warn,
+                              "Exception cleaning up inflight translation: {}",
+                              f.get_exception());
path: src/v/datalake/translation/partition_translator.cc
user: login: oleiman
body: oh, so even if discard returns an exceptional future, the expectation is that translation state is sufficiently reset? 
created_at: 2025-03-11T21:57:50Z
author_association: MEMBER
diff_hunk: @@ -211,7 +213,7 @@ def wait_for_translation_until_offset(self,
                                           topic,
                                           offset,
                                           partition=0,
-                                          timeout=30,
+                                          timeout=120,
path: tests/rptest/tests/datalake/datalake_services.py
user: login: andrwng
body: nit: how about just setting the cluster property wherever we set the commit interval? Waiting for longer (and this _is_ quite a while) seems like it leaves room for undesirable scheduling bugs to be ignored
created_at: 2025-03-11T21:59:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -322,6 +323,19 @@ ss::future<> partition_translator::init(
                     "[{}] Encountered exception in translation loop: {}",
                     ex,
                     _data_source->ntp());
+              })
+              .then([this] {
+                  // discard any inflight state and start from scratch
+                  return _translation_ctx->discard().then_wrapped(
+                    [this](ss::future<> f) {
+                        if (f.failed()) {
+                            vlog(
+                              _logger.warn,
+                              "Exception cleaning up inflight translation: {}",
+                              f.get_exception());
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: ya, the implementation calls translation_task::discard()&& which effectively means we destroy the exceptional translation_task state and create a new one in the next attempt. So we start from scratch after an exception.
created_at: 2025-03-12T02:20:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
path: src/v/datalake/translation/deps.cc
user: login: bharathv
body: thats how I started actually but the way we calculate lag with the heuristic and timestamp approximation today,  the window is automatically reset after checkpointing state, so effectively doesn't require a special notification to this interface.
created_at: 2025-03-12T03:09:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
+        return window_ended;
+    }
+
+private:
+    scheduling::clock::duration target_lag() const {
+        const auto& topic_cfg = _topics.get_topic_cfg(
+          model::topic_namespace_view{_partition->ntp()});
+        auto default_lag
+          = config::shard_local_cfg().iceberg_target_lag_ms.value();
+        if (!topic_cfg.has_value()) {
+            return std::chrono::duration_cast<scheduling::clock::duration>(
+              default_lag);
+        }
+        auto target_lag = topic_cfg->properties.iceberg_target_lag_ms.value_or(
+          default_lag);
+        return std::chrono::duration_cast<scheduling::clock::duration>(
+          target_lag);
+    }
+
+    void reset_to_next_window() { _current_window_end += target_lag(); }
path: src/v/datalake/translation/deps.cc
user: login: bharathv
body: ya just throw away code.
created_at: 2025-03-12T03:09:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -180,6 +182,17 @@ class translation_context {
      */
     virtual ss::future<> flush() = 0;
 
+    /**
+     * Returns the number of bytes that are flushed to the writer from the
+     * inflight translation.
+     */
+    virtual size_t flushed_bytes() const = 0;
+
+    /**
+     * Returns the last translated offset by the translator, if one exists.
+     */
+    virtual std::optional<kafka::offset> last_translated_offset() const = 0;
path: src/v/datalake/translation/deps.h
user: login: bharathv
body: moved
created_at: 2025-03-12T03:40:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,6 +24,14 @@ using namespace std::chrono_literals;
 // A simple utility to conditionally retry with backoff on failures.
 constexpr model::timeout_clock::duration wait_timeout = 5s;
 
+// Purposefully set to a low-ish value (instead of 64/128_MiB) until space
+// management is fully integrated. The low value means less concurrent bytes
+// accumulated across all translators at any point which translates to less
+// pressure on space management enforce it's eviction policies. 32_MiB is still
+// a reasonable default compared to what we had before, but will be bumped soon.
+// note: this is _per_ (partition) translator.
+static constexpr size_t cumulative_flushed_byte_limit = 32_MiB;
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: >  that we could have arbitrarily many partitions with inflight

Right, yes.
created_at: 2025-03-12T03:40:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,7 +213,7 @@ def wait_for_translation_until_offset(self,
                                           topic,
                                           offset,
                                           partition=0,
-                                          timeout=30,
+                                          timeout=120,
path: tests/rptest/tests/datalake/datalake_services.py
user: login: bharathv
body: actually this was not needed, remnant of some testing, undone.
created_at: 2025-03-12T04:30:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -577,4 +590,59 @@ translation_context::make_default_translation_context(
       std::move(probe));
 }
 
+// note: this is a temporary implementation to get the code compiling
+// to be filled with a proper implementation that tracks windows based
+// on batch timestamps + heuristics
+// Even the interface is temporary and subject to change depending on
+// the changes needed to track windows.
+class partition_lag_tracker : public translation_lag_tracker {
+public:
+    explicit partition_lag_tracker(
+      ss::lw_shared_ptr<cluster::partition> partition,
+      cluster::topic_table& topics)
+      : _partition(std::move(partition))
+      , _topics(topics)
+      , _current_window_end(scheduling::clock::now()) {
+        reset_to_next_window();
+    }
+
+    bool should_finish_inflight_translation() final {
+        auto window_ended = scheduling::clock::now() >= _current_window_end;
+        if (window_ended) {
+            reset_to_next_window();
+        }
path: src/v/datalake/translation/deps.cc
user: login: andrwng
body: >the way we calculate lag with the heuristic and timestamp approximation today, the window is automatically reset after checkpointing state

Yea I can see that, I don't love that the interface reflects the implementation detail. Not too important though, I guess we can change it as the heuristic evolves.
created_at: 2025-03-12T07:31:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,7 +213,7 @@ def wait_for_translation_until_offset(self,
                                           topic,
                                           offset,
                                           partition=0,
-                                          timeout=30,
+                                          timeout=120,
path: tests/rptest/tests/datalake/datalake_services.py
user: login: andrwng
body: Sure, same feedback for the rest of the ducktape changes. Why not just have them set the cluster properties? Every test basically expects that anyway by virtue of the commit interval being so low
created_at: 2025-03-12T07:34:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -197,65 +222,77 @@ ss::future<> partition_translator::translate_until_stopped() {
         // Update partition metrics. Note that last committed offset here
         // is NOT synchronized with outstanding commit operations. Therefore
         // if we reach this point before the most recent batch of files has
-        // been committed, the commit lag metric will be out of sync at least
-        // until 'wait_for_data' returns and we re-enter the loop.
+        // been committed, the commit lag metric will be out of sync at
+        // least until 'wait_for_data' returns and we re-enter the loop.
         _data_source->update_commit_lag(last_committed_offset);
         _data_source->update_translation_lag(last_translated_offset);
 
+        auto local_last_translated_offset = last_translated_offset;
+        if (
+          auto last_known_translated_offset
+          = _translation_ctx->last_translated_offset()) {
+            // A translation already in progress, start from where we left off.
+            local_last_translated_offset = last_known_translated_offset.value();
+        }
+
+        static constexpr auto data_wait_duration = 3s;
         // Wait until some data is ready to be translated.
         auto maybe_begin_offset
           = co_await _data_source->wait_for_data_to_translate(
-            last_translated_offset, _as);
+            local_last_translated_offset,
+            ss::lowres_clock::now() + data_wait_duration,
+            _as);
 
-        // if wait_for_data timed out (i.e. all trnaslatable records have
+        // if wait_for_data timed out (i.e. all translatable records have
         // been translated already), reenter the loop. this gives us an
         // opportunity to reconcile outstanding coordinator state, which is
         // helpful for keeping lag metrics up-to-date.
-        if (!maybe_begin_offset.has_value()) {
-            continue;
+        if (maybe_begin_offset) {
+            auto begin_offset = maybe_begin_offset.value();
+
+            // Notify the scheduler that there is some data to translate
+            _scheduler->notify_ready(id);
path: src/v/datalake/translation/partition_translator.cc
user: login: mmaslankaprv
body: is it ok to notify that we are ready for translation before finishing the previous one 
created_at: 2025-03-12T07:57:11Z
author_association: MEMBER
diff_hunk: @@ -197,65 +222,77 @@ ss::future<> partition_translator::translate_until_stopped() {
         // Update partition metrics. Note that last committed offset here
         // is NOT synchronized with outstanding commit operations. Therefore
         // if we reach this point before the most recent batch of files has
-        // been committed, the commit lag metric will be out of sync at least
-        // until 'wait_for_data' returns and we re-enter the loop.
+        // been committed, the commit lag metric will be out of sync at
+        // least until 'wait_for_data' returns and we re-enter the loop.
         _data_source->update_commit_lag(last_committed_offset);
         _data_source->update_translation_lag(last_translated_offset);
 
+        auto local_last_translated_offset = last_translated_offset;
+        if (
+          auto last_known_translated_offset
+          = _translation_ctx->last_translated_offset()) {
+            // A translation already in progress, start from where we left off.
+            local_last_translated_offset = last_known_translated_offset.value();
+        }
+
+        static constexpr auto data_wait_duration = 3s;
         // Wait until some data is ready to be translated.
         auto maybe_begin_offset
           = co_await _data_source->wait_for_data_to_translate(
-            last_translated_offset, _as);
+            local_last_translated_offset,
+            ss::lowres_clock::now() + data_wait_duration,
+            _as);
 
-        // if wait_for_data timed out (i.e. all trnaslatable records have
+        // if wait_for_data timed out (i.e. all translatable records have
         // been translated already), reenter the loop. this gives us an
         // opportunity to reconcile outstanding coordinator state, which is
         // helpful for keeping lag metrics up-to-date.
-        if (!maybe_begin_offset.has_value()) {
-            continue;
+        if (maybe_begin_offset) {
+            auto begin_offset = maybe_begin_offset.value();
+
+            // Notify the scheduler that there is some data to translate
+            _scheduler->notify_ready(id);
path: src/v/datalake/translation/partition_translator.cc
user: login: bharathv
body: no, its not. The call becomes a no-op, logs a warning and exits. A translator can only be in one state at a time. (scheduler expects a translator to be well behaved in some sense).
created_at: 2025-03-12T15:24:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -211,7 +213,7 @@ def wait_for_translation_until_offset(self,
                                           topic,
                                           offset,
                                           partition=0,
-                                          timeout=30,
+                                          timeout=120,
path: tests/rptest/tests/datalake/datalake_services.py
user: login: bharathv
body: Regarding rest of the changes

> tests/rptest/tests/datalake/recovery_mode_test.py

This is a special test that is running into some flakiness in coordinator API post restore and hitting backoff limits (hence a special bump in timeout)

The other changes could be removed too, probably (most of them were fiddling with timeouts to see what works and what doesn't. I'll merge this so Michal can rebase, I have another PR coming up soon, will cleanup test code a bit more.
created_at: 2025-03-12T15:35:28Z
author_association: CONTRIBUTOR

PR comments:
user: login: bharathv
created_at: 2025-03-10T19:21:23Z
body: /dt
author_association: CONTRIBUTOR
user: login: bharathv
created_at: 2025-03-10T19:37:19Z
body: /dt


author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-03-10T23:57:57Z
body: <!--TEST RESULTS-->
<!--bc2811b8-c5fa-4a00-a650-ea435376b6c3-->
<!--TEST RESULTS-->
<!--a3792d88-cbbd-4b8e-ae4e-15ab991fe57b-->
<!--TEST RESULTS-->
<!--9b487319-2c52-4e37-a2f9-03ce6669a7f8-->
<!--TEST RESULTS-->
<!--c6d1d62d-3dda-419f-95c2-186111f0ed5c-->
<!--title-->
## CI test results
<!--/title-->
<!--body-->
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62871">build#62871</a></summary>


| test_id | test_kind | job_url | test_status | passed |
|:----------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| translator_test_rpfixture.translator_test_rpfixture | unit | https://buildkite.com/redpanda/redpanda/builds/62871#01958191-5d0e-4f28-a2b0-3bd7333f037b | FAIL | 0/2 |
| translator_test_rpfixture.translator_test_rpfixture | unit | https://buildkite.com/redpanda/redpanda/builds/62871#01958191-5d0f-4563-8091-55e363dbd8a9 | FAIL | 0/2 |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62899">build#62899</a></summary>


| test_id | test_kind | job_url | test_status | passed |
|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4699-90af-4280bcc3d6bb | FLAKY | 1/2 |
| rptest.tests.consumer_group_test.ConsumerGroupTest.test_group_lag_metrics | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4699-90af-4280bcc3d6bb | FLAKY | 1/2 |
| rptest.tests.data_migrations_api_test.DataMigrationsApiTest.test_creating_and_listing_migrations | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4f34-ad3a-6545a1de043a | FLAKY | 1/2 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbe-462d-ac87-32494cebe060 | FLAKY | 1/18 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4699-90af-4280bcc3d6bb | FLAKY | 1/13 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbe-493d-bc06-ea00aea1d7be | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4f34-ad3a-6545a1de043a | FLAKY | 1/18 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbf-4e90-8514-df3be8d435eb | FLAKY | 1/4 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a5a-4836-a167-95641b5d7dd3 | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbf-46dd-999c-e16a77c70daa | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.NESSIE | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a5b-481c-beaa-2c01a93ebfcf | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbe-462d-ac87-32494cebe060 | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4699-90af-4280bcc3d6bb | FAIL | 0/20 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#01958427-5bbe-493d-bc06-ea00aea1d7be | FLAKY | 1/16 |
| rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4f34-ad3a-6545a1de043a | FLAKY | 1/19 |
| rptest.tests.partition_balancer_test.PartitionBalancerTest.test_unavailable_nodes | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a59-4699-90af-4280bcc3d6bb | FLAKY | 1/2 |
| rptest.tests.partition_movement_test.SIPartitionMovementTest.test_cross_shard.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS | ducktape | https://buildkite.com/redpanda/redpanda/builds/62899#0195842a-8a5a-4836-a167-95641b5d7dd3 | FLAKY | 1/3 |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62913">build#62913</a></summary>


| test_id | test_kind | job_url | test_status | passed |
|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.datalake.cluster_restore_test.DatalakeClusterRestoreTest.test_slow_tiered_storage_dlq.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP | ducktape | https://buildkite.com/redpanda/redpanda/builds/62913#019585cb-ac42-4f6c-bdf5-b5a037891745 | FLAKY | 1/2 |
| rptest.tests.partition_balancer_test.PartitionBalancerTest.test_full_nodes | ducktape | https://buildkite.com/redpanda/redpanda/builds/62913#019585cf-da23-462c-a9cf-3d24882ae3b5 | FLAKY | 1/2 |
| rptest.tests.partition_movement_test.SIPartitionMovementTest.test_shadow_indexing.num_to_upgrade=0.cloud_storage_type=CloudStorageType.ABS | ducktape | https://buildkite.com/redpanda/redpanda/builds/62913#019585cf-da23-462c-a9cf-3d24882ae3b5 | FLAKY | 1/2 |
</details>
<details>
<summary>test results on <a href="https://buildkite.com/redpanda/redpanda/builds/62966">build#62966</a></summary>


| test_id                                                                                                                                                                                                                                           | test_kind   | job_url                                                                                   | test_status   | passed   |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------|:------------------------------------------------------------------------------------------|:--------------|:---------|
| rptest.tests.datalake.batching_test.DatalakeBatchingTest.test_batching.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.catalog_type=CatalogType.REST_JDBC.expect_large_files=True                                       | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62966#01958914-33ca-4641-a0d6-e4500d367af0 | FLAKY         | 1/2      |
| rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=False.with_tiered_storage=False.with_iceberg=True.with_chunked_compaction=True.cloud_storage_type=CloudStorageType.S3 | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62966#01958914-33ca-4805-ba74-aa798a88b12d | FLAKY         | 1/2      |
| rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic                                                                                                                                                                   | ducktape    | https://buildkite.com/redpanda/redpanda/builds/62966#01958914-33ca-4641-a0d6-e4500d367af0 | FLAKY         | 1/2      |
</details>
<!--/body-->
author_association: COLLABORATOR
user: login: bharathv
created_at: 2025-03-11T01:47:35Z
body: /dt
author_association: CONTRIBUTOR
user: login: bharathv
created_at: 2025-03-11T05:58:56Z
body: /dt
author_association: CONTRIBUTOR
user: login: vbotbuildovich
created_at: 2025-03-11T09:43:40Z
body: <!--FAILURES SLASH COMMAND https://buildkite.com/redpanda/redpanda/builds/62899-->
<!--2585b467-d09a-4c2d-b1b3-6e26b97db82c-->
<!--98bc9e32-75d9-42d4-a993-1d88663bc8f5-->
<!--4b6936ae-3e51-449c-b197-1256028544f7-->
<!--0a669fbf-073e-4db8-8531-ccd144731f85-->
<!--eece30e7-06b5-4ee3-aac6-54f2fec7781c-->
<!--bd6c72f4-f26e-4c27-8abf-57d6fc074ec0-->
<!--title-->
## Retry command for [Build#62899](https://buildkite.com/redpanda/redpanda/builds/62899)
please wait until all jobs are finished before running the slash command
<!--/title-->
<!--body-->
```


/ci-repeat 1
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"catalog_type":"nessie","cloud_storage_type":1}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"catalog_type":"rest_hadoop","cloud_storage_type":1}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"catalog_type":"rest_hadoop","cloud_storage_type":1}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"catalog_type":"rest_jdbc","cloud_storage_type":1}
```
<!--/body-->
author_association: COLLABORATOR