Repository: scylladb/scylladb
Pull request #23727: backup: Optimize S3 throughput with shard-based upload
Author: login: kreuzerkrieg
Labels: name: P1
description: Urgent
name: backport/none
description: Backport is not required
name: promoted-to-master
name: area/backup_and_restore
Created: 2025-04-14T13:25:20Z | Merged: 2025-04-19T18:08:40Z

This PR enhances S3 throughput by leveraging every available shard to upload backup files concurrently. By distributing the load across multiple shards, we significantly improve the upload performance. Each shard retrieves an SSTable and processes its files sequentially, ensuring efficient, file-by-file uploads.

To prevent uncontrolled fiber creation and potential resource exhaustion, the backup task employs a directory semaphore from the sstables_manager. This mechanism helps regulate concurrency at the directory level, ensuring stable and predictable performance during large-scale backup operations.

Refs #22460
fixes: #22520

## Quick perf comparison

```
===========================================
 Release build, master, smp-16, mem-32GiB
 Bytes: 2342880184, backup time: 9.51 s
===========================================
 Release build, this PR, smp-16, mem-32GiB
 Bytes: 2342891015, backup time: 1.23 s
===========================================
```
Looks like it is faster at least x7.7

No backport needed since it (native backup) is still unused functionality



Diff:
diff --git a/db/snapshot-ctl.cc b/db/snapshot-ctl.cc
index dd09cd488bd0..f8ebe47f4288 100644
--- a/db/snapshot-ctl.cc
+++ b/db/snapshot-ctl.cc
@@ -142,7 +142,6 @@ future<tasks::task_id> snapshot_ctl::start_backup(sstring endpoint, sstring buck
     }
 
     co_await coroutine::switch_to(_config.backup_sched_group);
-    auto cln = _storage_manager.get_endpoint_client(endpoint);
     snap_log.info("Backup sstables from {}({}) to {}", keyspace, snapshot_name, endpoint);
     auto global_table = co_await get_table_on_all_shards(_db, keyspace, table);
     auto& storage_options = global_table->get_storage_options();
@@ -172,7 +171,7 @@ future<tasks::task_id> snapshot_ctl::start_backup(sstring endpoint, sstring buck
                 sstables::snapshots_dir /
                 std::string_view(snapshot_name));
     auto task = co_await _task_manager_module->make_and_start_task<::db::snapshot::backup_task_impl>(
-        {}, *this, std::move(cln), std::move(bucket), std::move(prefix), keyspace, dir, global_table->schema()->id(), move_files);
+        {}, *this, _storage_manager.container(), std::move(endpoint), std::move(bucket), std::move(prefix), keyspace, dir, global_table->schema()->id(), move_files);
     co_return task->id();
 }
 
diff --git a/db/snapshot/backup_task.cc b/db/snapshot/backup_task.cc
index 75bf5c61288b..f8bd235c6e44 100644
--- a/db/snapshot/backup_task.cc
+++ b/db/snapshot/backup_task.cc
@@ -31,7 +31,8 @@ namespace db::snapshot {
 
 backup_task_impl::backup_task_impl(tasks::task_manager::module_ptr module,
                                    snapshot_ctl& ctl,
-                                   shared_ptr<s3::client> client,
+                                   sharded<sstables::storage_manager>& sstm,
+                                   sstring endpoint,
                                    sstring bucket,
                                    sstring prefix,
                                    sstring ks,
@@ -40,7 +41,8 @@ backup_task_impl::backup_task_impl(tasks::task_manager::module_ptr module,
                                    bool move_files) noexcept
     : tasks::task_manager::task::impl(module, tasks::task_id::create_random_id(), 0, "node", ks, "", "", tasks::task_id::create_null_id())
     , _snap_ctl(ctl)
-    , _client(std::move(client))
+    , _sstm(sstm)
+    , _endpoint(std::move(endpoint))
     , _bucket(std::move(bucket))
     , _prefix(std::move(prefix))
     , _snapshot_dir(std::move(snapshot_dir))
@@ -62,18 +64,23 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    auto p = co_await _sstm.map_reduce0(
+        [this](const auto&) {
+            return _progress_per_shard[this_shard_id()];
+        }, s3::upload_progress(), std::plus<>());
+    co_return tasks::task_manager::task::progress{
+        .completed = p.uploaded,
+        .total = p.total,
+    };
 }
 
 tasks::is_user_task backup_task_impl::is_user_task() const noexcept {
     return tasks::is_user_task::yes;
 }
 
-future<> backup_task_impl::upload_component(sstring name) {
-    auto component_name = _snapshot_dir / name;
-    auto destination = fmt::format("/{}/{}/{}", _bucket, _prefix, name);
+future<> backup_task_impl::worker::upload_component(sstring name) {
+    auto component_name = _task._snapshot_dir / name;
+    auto destination = fmt::format("/{}/{}/{}", _task._bucket, _task._prefix, name);
     snap_log.trace("Upload {} to {}", component_name.native(), destination);
 
     // Start uploading in the background. The caller waits for these fibers
@@ -82,7 +89,7 @@ future<> backup_task_impl::upload_component(sstring name) {
     //  - s3::client::claim_memory semaphore
     //  - http::client::max_connections limitation
     try {
-        co_await _client->upload_file(component_name, destination, _progress, &_as);
+        co_await _client->upload_file(component_name, destination, _task._progress_per_shard[this_shard_id()], &_as);
     } catch (const abort_requested_exception&) {
         snap_log.info("Upload aborted per requested: {}", component_name.native());
         throw;
@@ -91,7 +98,7 @@ future<> backup_task_impl::upload_component(sstring name) {
         throw;
     }
 
-    if (!_remove_on_uploaded) {
+    if (!_task._remove_on_uploaded) {
         co_return;
     }
 
@@ -118,12 +125,25 @@ future<> backup_task_impl::do_backup() {
     _backup_shard = this_shard_id();
     co_await _sharded_worker.start(std::ref(_snap_ctl.db()), _table_id, std::ref(*this));
 
+    gate abort_gate;
+    auto abort_sub = _as.subscribe([&] () noexcept {
+        if (auto gh = abort_gate.try_hold()) {
+            std::ignore = _sharded_worker.invoke_on_all([] (worker& m) {
+                m.abort();
+            }).finally([gh = std::move(gh)] {});
+        }
+    });
+
     try {
-        co_await uploads_worker();
+        co_await _sharded_worker.invoke_on_all([](worker& m) {
+            return m.start_uploading();
+        });
     } catch (...) {
         _ex = std::current_exception();
     }
 
+    co_await abort_gate.close();
+    abort_sub = {};
     co_await _sharded_worker.stop();
 
     if (_ex) {
@@ -175,27 +195,30 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::start_uploading() {
+    named_gate uploads(format("do_backup::uploads({})", _task._snapshot_dir));
 
     try {
         while (!_ex) {
             auto gh = uploads.hold();
-            auto units = co_await _sharded_worker.local().manager().dir_semaphore().get_units(1, _as);
+            auto units = co_await _manager.dir_semaphore().get_units(1, _as);
 
             // Pre-upload break point. For testing abort in actual s3 client usage.
             co_await utils::get_local_injector().inject("backup_task_pre_upload", utils::wait_for_message(std::chrono::minutes(2)));
 
-            auto name_opt = dequeue();
+            auto name_opt = co_await smp::submit_to(_task._backup_shard, [this] () {
+                return _task.dequeue();
+            });
             if (!name_opt) {
                 break;
             }
-            // okay to drop future since uploads is always closed before exiting the function
-            std::ignore = backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
+            // okay to drop future since async_gate is always closed before stopping
+            std::ignore =
+                backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
             co_await coroutine::maybe_yield();
             co_await utils::get_local_injector().inject("backup_task_pause", utils::wait_for_message(std::chrono::minutes(2)));
-            if (impl::_as.abort_requested()) {
-                _ex = impl::_as.abort_requested_exception_ptr();
+            if (_as.abort_requested()) {
+                _ex = _as.abort_requested_exception_ptr();
                 break;
             }
         }
@@ -210,7 +233,7 @@ future<> backup_task_impl::uploads_worker() {
     }
 }
 
-future<> backup_task_impl::backup_file(sstring name, upload_permit permit) {
+future<> backup_task_impl::worker::backup_file(sstring name, upload_permit permit) {
     try {
         co_await upload_component(name);
     } catch (...) {
@@ -269,10 +292,15 @@ void backup_task_impl::on_sstable_deletion(sstables::generation_type gen) {
 backup_task_impl::worker::worker(const replica::database& db, table_id t, backup_task_impl& task)
     : _manager(db.get_sstables_manager(*db.find_schema(t)))
     , _task(task)
+    , _client(task._sstm.local().get_endpoint_client(task._endpoint))
 {
     _manager.subscribe(*this);
 }
 
+void backup_task_impl::worker::abort() {
+    _as.request_abort();
+}
+
 future<> backup_task_impl::worker::deleted_sstable(sstables::generation_type gen) const {
     // The notification is called for any sstable, so `gen` may belong
     // to another table, or to an sstable that was created after the snapshot
diff --git a/db/snapshot/backup_task.hh b/db/snapshot/backup_task.hh
index 1fd2059d051e..ee0745284e79 100644
--- a/db/snapshot/backup_task.hh
+++ b/db/snapshot/backup_task.hh
@@ -17,6 +17,8 @@
 
 #include <seastar/core/gate.hh>
 #include <seastar/core/semaphore.hh>
+#include <seastar/core/shard_id.hh>
+#include <seastar/core/sharded.hh>
 
 #include "utils/s3/client_fwd.hh"
 #include "utils/small_vector.hh"
@@ -31,6 +33,7 @@ namespace replica {
 
 namespace sstables {
 class sstables_manager;
+class storage_manager;
 }
 
 namespace db {
@@ -40,14 +43,14 @@ namespace snapshot {
 
 class backup_task_impl : public tasks::task_manager::task::impl {
     snapshot_ctl& _snap_ctl;
-    shared_ptr<s3::client> _client;
+    sharded<sstables::storage_manager>& _sstm;
+    sstring _endpoint;
     sstring _bucket;
     sstring _prefix;
     std::filesystem::path _snapshot_dir;
     table_id _table_id;
     bool _remove_on_uploaded;
     tasks::task_manager::task::progress _total_progress;
-    s3::upload_progress _progress;
 
     std::exception_ptr _ex;
     std::vector<sstring> _files;
@@ -61,28 +64,35 @@ class backup_task_impl : public tasks::task_manager::task::impl {
     class worker : sstables::sstables_manager_event_handler {
         sstables::sstables_manager& _manager;
         backup_task_impl& _task;
+        shared_ptr<s3::client> _client;
+        abort_source _as;
+        std::exception_ptr _ex;
 
     public:
         worker(const replica::database& db, table_id t, backup_task_impl& task);
 
+        future<> start_uploading();
+
+        void abort();
+
         sstables::sstables_manager& manager() const noexcept {
             return _manager;
         }
 
     private:
         virtual future<> deleted_sstable(sstables::generation_type gen) const override;
+        struct upload_permit {
+            gate::holder gh;
+            semaphore_units<> units;
+        };
+        future<> backup_file(sstring name, upload_permit permit);
+        future<> upload_component(sstring name);
     };
     sharded<worker> _sharded_worker;
+    std::vector<s3::upload_progress> _progress_per_shard{smp::count};
 
     future<> do_backup();
-    future<> upload_component(sstring name);
     future<> process_snapshot_dir();
-    future<> uploads_worker();
-    struct upload_permit {
-        gate::holder gh;
-        semaphore_units<> units;
-    };
-    future<> backup_file(sstring name, upload_permit permit);
     // Returns a disengaged optional when done
     std::optional<std::string> dequeue();
     void dequeue_sstable();
@@ -94,7 +104,8 @@ protected:
 public:
     backup_task_impl(tasks::task_manager::module_ptr module,
                      snapshot_ctl& ctl,
-                     shared_ptr<s3::client> cln,
+                     sharded<sstables::storage_manager>& sstm,
+                     sstring endpoint,
                      sstring bucket,
                      sstring prefix,
                      sstring ks,
diff --git a/test/cluster/object_store/test_backup.py b/test/cluster/object_store/test_backup.py
index 6b2185731642..ba9e350fef1e 100644
--- a/test/cluster/object_store/test_backup.py
+++ b/test/cluster/object_store/test_backup.py
@@ -148,8 +148,34 @@ async def test_backup_to_non_existent_bucket(manager: ManagerClient, s3_server):
     status = await manager.api.wait_task(server.ip_addr, tid)
     assert status is not None
     assert status['state'] == 'failed'
+    assert 'S3 request failed. Code: 15. Reason: Access Denied.' in status['error']
 
 
+async def test_backup_to_non_existent_endpoint(manager: ManagerClient, s3_server):
+    '''backup should fail if the endpoint is invalid/inaccessible'''
+
+    objconf = MinioServer.create_conf(s3_server.address, s3_server.port, s3_server.region)
+    cfg = {'enable_user_defined_functions': False,
+           'object_storage_endpoints': objconf,
+           'experimental_features': ['keyspace-storage-options'],
+           'task_ttl_in_seconds': 300
+           }
+    cmd = ['--logger-log-level', 'snapshots=trace:task_manager=trace']
+    server = await manager.server_add(config=cfg, cmdline=cmd)
+    ks, cf = await prepare_snapshot_for_backup(manager, server)
+
+    workdir = await manager.server_get_workdir(server.server_id)
+    cf_dir = os.listdir(f'{workdir}/data/{ks}')[0]
+    files = set(os.listdir(f'{workdir}/data/{ks}/{cf_dir}/snapshots/backup'))
+    assert len(files) > 0
+
+    prefix = f'{cf}/backup'
+    tid = await manager.api.backup(server.ip_addr, ks, cf, 'backup', "does_not_exist", s3_server.bucket_name, prefix)
+    status = await manager.api.wait_task(server.ip_addr, tid)
+    assert status is not None
+    assert status['state'] == 'failed'
+    assert status['error'] == 'std::invalid_argument (endpoint does_not_exist not found)'
+
 async def do_test_backup_abort(manager: ManagerClient, s3_server,
                                breakpoint_name, min_files, max_files = None):
     '''helper for backup abort testing'''
@@ -187,6 +213,7 @@ async def do_test_backup_abort(manager: ManagerClient, s3_server,
     status = await manager.api.wait_task(server.ip_addr, tid)
     print(f'Status: {status}')
     assert (status is not None) and (status['state'] == 'failed')
+    assert "seastar::abort_requested_exception (abort requested)" in status['error']
 
     objects = set(o.key for o in get_s3_resource(s3_server).Bucket(s3_server.bucket_name).objects.all())
     uploaded_count = 0
diff --git a/utils/s3/client_fwd.hh b/utils/s3/client_fwd.hh
index f98cfb482e2c..556796a7e1c7 100644
--- a/utils/s3/client_fwd.hh
+++ b/utils/s3/client_fwd.hh
@@ -15,5 +15,6 @@ class client;
 struct upload_progress {
     size_t total = 0;
     size_t uploaded = 0;
+    upload_progress operator+(const upload_progress& other) const { return {total + other.total, uploaded + other.uploaded}; }
 };
 }


Reviews:
user: login: bhalevy
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-14T13:31:57Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T14:24:58Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T14:26:41Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T14:30:07Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T14:34:32Z
user: login: bhalevy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T08:49:38Z
user: login: bhalevy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T08:52:38Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:55:11Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:57:14Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:59:07Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:03:07Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:04:00Z
user: login: bhalevy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T14:05:44Z
user: login: bhalevy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T14:08:43Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T09:45:04Z
user: login: avikivity
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T09:54:00Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T09:59:10Z
user: login: avikivity
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T10:04:08Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T11:36:28Z
user: login: bhalevy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T11:39:57Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T12:30:11Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T12:30:28Z
user: login: Deexie
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T12:54:11Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T12:59:58Z
user: login: xemul
body: LGTM, let's see what CI thinks
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:00:56Z
user: login: xemul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:28:32Z

Review comments:
diff_hunk: @@ -110,7 +110,7 @@ private:
     seastar::rwlock _lock;
     seastar::gate _ops;
     shared_ptr<snapshot::task_manager_module> _task_manager_module;
-    sstables::storage_manager& _storage_manager;
+    sharded<sstables::storage_manager>& _storage_manager;
path: db/snapshot-ctl.hh
user: login: xemul
body: storage_manager is peering sharded service, you can call `_storage_manager.container()` at any time to get the sharded<...> reference
created_at: 2025-04-14T14:24:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,17 +65,30 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
     auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
+    // _total_progress.completed before _sharded_worker is stopped
path: db/snapshot/backup_task.cc
user: login: xemul
body: A verb is missing. Like "_total_progress.completed _is updated_ before _shared_worker is stopped ..." or something. But to be honest, I don't yet understand what's happening with progress accounting in this PR
created_at: 2025-04-14T14:26:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,8 +133,30 @@ future<> backup_task_impl::do_backup() {
     _backup_shard = this_shard_id();
     co_await _sharded_worker.start(std::ref(_snap_ctl.db()), _table_id, std::ref(*this));
 
+    gate abort_gate;
+    auto abort_sub = _as.subscribe([&] () noexcept {
+        if (auto gh = abort_gate.try_hold()) {
+            std::ignore = _sharded_worker.invoke_on_all([] (worker& m) {
+                m.abort();
+            }).finally([gh = std::move(gh)] {});
+        }
+    });
+
+    // Errors from `start_uploading` are retrieved by `done`
+    co_await _sharded_worker.invoke_on_all([] (worker& m) {
+        return m.start_uploading();
+    });
+
     try {
-        co_await uploads_worker();
+        co_await _sharded_worker.invoke_on_all([] (worker& m) {
+            return m.done();
+        });
+
+        co_await abort_gate.close();
path: db/snapshot/backup_task.cc
user: login: xemul
body: Shouldn't abort_gate be close()-ed on exception too?
created_at: 2025-04-14T14:30:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,8 +133,30 @@ future<> backup_task_impl::do_backup() {
     _backup_shard = this_shard_id();
     co_await _sharded_worker.start(std::ref(_snap_ctl.db()), _table_id, std::ref(*this));
 
+    gate abort_gate;
+    auto abort_sub = _as.subscribe([&] () noexcept {
+        if (auto gh = abort_gate.try_hold()) {
+            std::ignore = _sharded_worker.invoke_on_all([] (worker& m) {
+                m.abort();
+            }).finally([gh = std::move(gh)] {});
+        }
+    });
+
+    // Errors from `start_uploading` are retrieved by `done`
+    co_await _sharded_worker.invoke_on_all([] (worker& m) {
+        return m.start_uploading();
+    });
+
     try {
-        co_await uploads_worker();
+        co_await _sharded_worker.invoke_on_all([] (worker& m) {
+            return m.done();
path: db/snapshot/backup_task.cc
user: login: xemul
body: Why the pattern here is

```c++
    co_await sharded_worker.invoke_on_all(&worker::start);
    co_await sharded_worker.invoke_on_all(&worker::done);
```

isn't it enough if `worker::start()` just doesn't resolve until it's done, so that the caller goes with the single co_await invoke-on-all?
created_at: 2025-04-14T14:34:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,8 +133,30 @@ future<> backup_task_impl::do_backup() {
     _backup_shard = this_shard_id();
     co_await _sharded_worker.start(std::ref(_snap_ctl.db()), _table_id, std::ref(*this));
 
+    gate abort_gate;
+    auto abort_sub = _as.subscribe([&] () noexcept {
+        if (auto gh = abort_gate.try_hold()) {
+            std::ignore = _sharded_worker.invoke_on_all([] (worker& m) {
+                m.abort();
+            }).finally([gh = std::move(gh)] {});
+        }
+    });
+
+    // Errors from `start_uploading` are retrieved by `done`
+    co_await _sharded_worker.invoke_on_all([] (worker& m) {
+        return m.start_uploading();
+    });
+
     try {
-        co_await uploads_worker();
+        co_await _sharded_worker.invoke_on_all([] (worker& m) {
+            return m.done();
path: db/snapshot/backup_task.cc
user: login: bhalevy
body: It should be possible to return non-ready future from start_uploading (and rename it to e.g. `upload`). 
I think I did it this way since start_uploading was initially launched in the (synchronous) constructor.

created_at: 2025-04-15T08:49:38Z
author_association: MEMBER
diff_hunk: @@ -118,8 +133,30 @@ future<> backup_task_impl::do_backup() {
     _backup_shard = this_shard_id();
     co_await _sharded_worker.start(std::ref(_snap_ctl.db()), _table_id, std::ref(*this));
 
+    gate abort_gate;
+    auto abort_sub = _as.subscribe([&] () noexcept {
+        if (auto gh = abort_gate.try_hold()) {
+            std::ignore = _sharded_worker.invoke_on_all([] (worker& m) {
+                m.abort();
+            }).finally([gh = std::move(gh)] {});
+        }
+    });
+
+    // Errors from `start_uploading` are retrieved by `done`
+    co_await _sharded_worker.invoke_on_all([] (worker& m) {
+        return m.start_uploading();
+    });
+
     try {
-        co_await uploads_worker();
+        co_await _sharded_worker.invoke_on_all([] (worker& m) {
+            return m.done();
+        });
+
+        co_await abort_gate.close();
path: db/snapshot/backup_task.cc
user: login: bhalevy
body: yes, if shard upload failed for some other reasons than abort, we should close the abort gate before stopping the _sharded_worker.
created_at: 2025-04-15T08:52:38Z
author_association: MEMBER
diff_hunk: @@ -62,18 +64,28 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_return tasks::task_manager::task::progress{
+        .completed = _total_upload_progress.uploaded,
+        .total = _total_upload_progress.total,
+    };
+}
+
+future<> backup_task_impl::update_total_progress() {
+    if (!_sharded_worker.local_is_initialized()) {
+        _total_upload_progress = {};
+    }
+    _total_upload_progress = co_await _sharded_worker.map_reduce0([](const auto& worker) {
path: db/snapshot/backup_task.cc
user: login: xemul
body: This makes little sense in the light of previous check. According to the check `_sharded_worker` can be not initialized here, but if this happens it's invalid to call `map_reduce()` on it
created_at: 2025-04-16T13:55:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -175,27 +201,29 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::uploads_worker() {
path: db/snapshot/backup_task.cc
user: login: xemul
body: Why not name this method `backup_task_impl::worker::start_uploading()`? The method of the same name exists, it's API one and it _just_ call this one anyway
created_at: 2025-04-16T13:57:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -61,40 +64,55 @@ class backup_task_impl : public tasks::task_manager::task::impl {
     class worker : sstables::sstables_manager_event_handler {
         sstables::sstables_manager& _manager;
         backup_task_impl& _task;
+        shared_ptr<s3::client> _client;
+        gate _uploads;
path: db/snapshot/backup_task.hh
user: login: xemul
body: This field is unused, is it?
created_at: 2025-04-16T13:59:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +64,28 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_return tasks::task_manager::task::progress{
path: db/snapshot/backup_task.cc
user: login: xemul
body: But `_total_upload_progress` is only changed in `update_total_progress()` which, in turn, is only called once _after_ all workers are done. So we'll see zeroes here throughout the whole uploading. What am I missing?
created_at: 2025-04-16T14:03:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,7 +171,7 @@ future<tasks::task_id> snapshot_ctl::start_backup(sstring endpoint, sstring buck
                 sstables::snapshots_dir /
                 std::string_view(snapshot_name));
     auto task = co_await _task_manager_module->make_and_start_task<::db::snapshot::backup_task_impl>(
-        {}, *this, std::move(cln), std::move(bucket), std::move(prefix), keyspace, dir, global_table->schema()->id(), move_files);
+        {}, *this, _storage_manager.container(), std::move(endpoint), std::move(bucket), std::move(prefix), keyspace, dir, global_table->schema()->id(), move_files);
path: db/snapshot-ctl.cc
user: login: xemul
body: nit (well, a tempting nit): Changing backup_task_impl to accept sharded<storage_manager> ref and endpoint sstring can be made as a separate preparation patch
created_at: 2025-04-16T14:04:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +64,28 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_return tasks::task_manager::task::progress{
+        .completed = _total_upload_progress.uploaded,
+        .total = _total_upload_progress.total,
+    };
+}
+
+future<> backup_task_impl::update_total_progress() {
+    if (!_sharded_worker.local_is_initialized()) {
+        _total_upload_progress = {};
+    }
+    _total_upload_progress = co_await _sharded_worker.map_reduce0([](const auto& worker) {
path: db/snapshot/backup_task.cc
user: login: bhalevy
body: good catch, it should be in the else branch
created_at: 2025-04-16T14:05:44Z
author_association: MEMBER
diff_hunk: @@ -175,27 +201,29 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::uploads_worker() {
path: db/snapshot/backup_task.cc
user: login: bhalevy
body: Or call `uploads_worker` above instead of `start_uploading`
created_at: 2025-04-16T14:08:43Z
author_association: MEMBER
diff_hunk: @@ -178,27 +202,32 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::start_uploading() {
+    named_gate uploads(format("do_backup::uploads({})", _task._snapshot_dir));
 
     try {
         while (!_ex) {
             auto gh = uploads.hold();
-            auto units = co_await _sharded_worker.local().manager().dir_semaphore().get_units(1, _as);
+            auto units = co_await _manager.dir_semaphore().get_units(1, _as);
 
             // Pre-upload break point. For testing abort in actual s3 client usage.
             co_await utils::get_local_injector().inject("backup_task_pre_upload", utils::wait_for_message(std::chrono::minutes(2)));
 
-            auto name_opt = dequeue();
+            auto name_opt = co_await smp::submit_to(_task._backup_shard, [this] () {
+                return _task.dequeue();
+            });
             if (!name_opt) {
                 break;
             }
-            // okay to drop future since uploads is always closed before exiting the function
-            std::ignore = backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
+            // okay to drop future since async_gate is always closed before stopping
+            std::ignore =
+                backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units))).then([this] {
+                    return _task.update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: This is horrible. Instead of bouncing to task shard once and updating the progress with two values in-place, you cause the task shard bounce back to _all_ other shards to collect the overall progress with map_reduce0
created_at: 2025-04-17T09:45:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -178,27 +202,32 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::start_uploading() {
+    named_gate uploads(format("do_backup::uploads({})", _task._snapshot_dir));
 
     try {
         while (!_ex) {
             auto gh = uploads.hold();
-            auto units = co_await _sharded_worker.local().manager().dir_semaphore().get_units(1, _as);
+            auto units = co_await _manager.dir_semaphore().get_units(1, _as);
 
             // Pre-upload break point. For testing abort in actual s3 client usage.
             co_await utils::get_local_injector().inject("backup_task_pre_upload", utils::wait_for_message(std::chrono::minutes(2)));
 
-            auto name_opt = dequeue();
+            auto name_opt = co_await smp::submit_to(_task._backup_shard, [this] () {
+                return _task.dequeue();
+            });
             if (!name_opt) {
                 break;
             }
-            // okay to drop future since uploads is always closed before exiting the function
-            std::ignore = backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
+            // okay to drop future since async_gate is always closed before stopping
+            std::ignore =
+                backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units))).then([this] {
+                    return _task.update_total_progress();
path: db/snapshot/backup_task.cc
user: login: avikivity
body: It's better for the one who is asking for progress to bounce, since it's rare for progress to be queried.
created_at: 2025-04-17T09:54:00Z
author_association: MEMBER
diff_hunk: @@ -178,27 +202,32 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::start_uploading() {
+    named_gate uploads(format("do_backup::uploads({})", _task._snapshot_dir));
 
     try {
         while (!_ex) {
             auto gh = uploads.hold();
-            auto units = co_await _sharded_worker.local().manager().dir_semaphore().get_units(1, _as);
+            auto units = co_await _manager.dir_semaphore().get_units(1, _as);
 
             // Pre-upload break point. For testing abort in actual s3 client usage.
             co_await utils::get_local_injector().inject("backup_task_pre_upload", utils::wait_for_message(std::chrono::minutes(2)));
 
-            auto name_opt = dequeue();
+            auto name_opt = co_await smp::submit_to(_task._backup_shard, [this] () {
+                return _task.dequeue();
+            });
             if (!name_opt) {
                 break;
             }
-            // okay to drop future since uploads is always closed before exiting the function
-            std::ignore = backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
+            // okay to drop future since async_gate is always closed before stopping
+            std::ignore =
+                backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units))).then([this] {
+                    return _task.update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: Agree, another option is to call map_reduce inside `get_progress()`. Just need to handle the case when workers are already dead (or still not alive)
created_at: 2025-04-17T09:59:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -178,27 +202,32 @@ future<> backup_task_impl::process_snapshot_dir() {
     }
 }
 
-future<> backup_task_impl::uploads_worker() {
-    named_gate uploads(format("do_backup::uploads({})", _snapshot_dir));
+future<> backup_task_impl::worker::start_uploading() {
+    named_gate uploads(format("do_backup::uploads({})", _task._snapshot_dir));
 
     try {
         while (!_ex) {
             auto gh = uploads.hold();
-            auto units = co_await _sharded_worker.local().manager().dir_semaphore().get_units(1, _as);
+            auto units = co_await _manager.dir_semaphore().get_units(1, _as);
 
             // Pre-upload break point. For testing abort in actual s3 client usage.
             co_await utils::get_local_injector().inject("backup_task_pre_upload", utils::wait_for_message(std::chrono::minutes(2)));
 
-            auto name_opt = dequeue();
+            auto name_opt = co_await smp::submit_to(_task._backup_shard, [this] () {
+                return _task.dequeue();
+            });
             if (!name_opt) {
                 break;
             }
-            // okay to drop future since uploads is always closed before exiting the function
-            std::ignore = backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units)));
+            // okay to drop future since async_gate is always closed before stopping
+            std::ignore =
+                backup_file(std::move(*name_opt), upload_permit(std::move(gh), std::move(units))).then([this] {
+                    return _task.update_total_progress();
path: db/snapshot/backup_task.cc
user: login: avikivity
body: They can unlink themselves when they're done.
created_at: 2025-04-17T10:04:08Z
author_association: MEMBER
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: This is called via API, and `co_await _sharded_worker.stop();` is called by task itself. If it happens in parallel, won't `update_total_progress()`'s call of map_reduce0 step on stopping `_sharded_worker` thing?
created_at: 2025-04-17T11:36:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: bhalevy
body: Hmm, to solve this "by the book" we may need sub-tasks for each shard.
Right @Deexie?
Otherwise, we'd need a gate held by the api path (prior to map_reduce) and be closed before the sharded worker is stopped.
created_at: 2025-04-17T11:39:57Z
author_association: MEMBER
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: There's another approach. The sstable_loader code also has sharded progress, and it solves this by carefully maintaining state of it, keeping the final map-reduced value when its stopped
created_at: 2025-04-17T12:30:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: But maybe there's a bug too
created_at: 2025-04-17T12:30:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: Deexie
body: I just checked and yes, there is the same bug there:

https://github.com/scylladb/scylladb/blob/84d4af1f0ef67bdbdcc49c4f5d4f53b7ef14ea81/sstables_loader.cc#L638-L668

if API invoked get_progress preempts in line 656 and release_resources() is started
created_at: 2025-04-17T12:54:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: @bhalevy , `std::vector<progress>` as in the recent version should work as well (not very cache friendly, but it's not updated that often)
created_at: 2025-04-17T12:59:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -65,18 +64,29 @@ tasks::is_abortable backup_task_impl::is_abortable() const noexcept {
 }
 
 future<tasks::task_manager::task::progress> backup_task_impl::get_progress() const {
-    auto ret = _total_progress;
-    ret.completed = _progress.uploaded;
-    co_return ret;
+    co_await update_total_progress();
path: db/snapshot/backup_task.cc
user: login: xemul
body: https://github.com/scylladb/scylladb/issues/23801 not to forget
created_at: 2025-04-17T14:28:31Z
author_association: CONTRIBUTOR

PR comments:
user: login: bhalevy
created_at: 2025-04-14T13:31:49Z
body: Can we please run a SCT test to compare backup performance under load before and after the change?
Let's see that the change behaves well in practice.

I hope that this change may also improve p99 latency since the backup load would be more evenly spread on all shards, but if the increased latency is caused by the network bandwidth used for backup, out-competing user queries coordinator/replica traffic, then it won't affect that much.

This should not block merging this PR though.

author_association: MEMBER
user: login: regevran
created_at: 2025-04-14T13:43:43Z
body: > Can we please run a SCT test to compare backup performance under load before and after the change?

The plan is to re-do performance tests only after all relevant PRs (including this one) are merged to both scylla and scylla-manager.
I prefer not to disrupt current effort to get the feature to be ready (correctness wise) and then focus of performance.
author_association: CONTRIBUTOR
user: login: scylladb-promoter
created_at: 2025-04-14T23:23:12Z
body: #### :green_circle: CI State: SUCCESS

:white_check_mark: - [Framework test](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16759/)
:white_check_mark: - [Build](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16759/)
:white_check_mark: - [Unit Tests Custom](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16759/)
**The following new/updated tests ran 100 times for each mode:**
:small_blue_diamond: cluster/object_store/test_backup
:white_check_mark: - [dtest with consistent topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-consistent-topology-changes/7930/)
:white_check_mark: - [dtest with tablets](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-tablets/4755/)
:white_check_mark: - [dtest with gossip topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release/15982/)
:white_check_mark: - [Unit Tests](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16759/)
#### Build Details:
- Duration: 9 hr 38 min
- Builder: spider1.cloudius-systems.com

author_association: CONTRIBUTOR
user: login: avikivity
created_at: 2025-04-15T11:46:49Z
body: Please enhance the cover letter with an explanation of the algorithm and before/after test.
author_association: MEMBER
user: login: kreuzerkrieg
created_at: 2025-04-15T12:22:51Z
body: > Please enhance the cover letter with an explanation of the algorithm and before/after test.

Do you  mean perf tests?
author_association: CONTRIBUTOR
user: login: scylladb-promoter
created_at: 2025-04-15T20:31:43Z
body: #### :green_circle: CI State: SUCCESS

:white_check_mark: - [Framework test](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16783/)
:white_check_mark: - [Build](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16783/)
:white_check_mark: - [Unit Tests Custom](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16783/)
**The following new/updated tests ran 100 times for each mode:**
:small_blue_diamond: cluster/object_store/test_backup
:white_check_mark: - [dtest with consistent topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-consistent-topology-changes/7959/)
:white_check_mark: - [dtest with gossip topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release/16008/)
:white_check_mark: - [dtest with tablets](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-tablets/4780/)
:white_check_mark: - [Unit Tests](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16783/)
#### Build Details:
- Duration: 10 hr
- Builder: spider3.cloudius-systems.com

author_association: CONTRIBUTOR
user: login: mykaul
created_at: 2025-04-16T05:32:29Z
body: @xemul - please review. 
author_association: CONTRIBUTOR
user: login: avikivity
created_at: 2025-04-16T17:49:29Z
body: > > Please enhance the cover letter with an explanation of the algorithm and before/after test.
> 
> Do you mean perf tests?

A test which will allow me or a user to say "this operation was improved by a factor of X". Could be `nodetool backup` or something.
author_association: MEMBER
user: login: kreuzerkrieg
created_at: 2025-04-16T18:09:41Z
body: > > > Please enhance the cover letter with an explanation of the algorithm and before/after test.
> > 
> > 
> > Do you mean perf tests?
> 
> A test which will allow me or a user to say "this operation was improved by a factor of X". Could be `nodetool backup` or something.

please note
https://github.com/scylladb/scylladb/pull/23727#issuecomment-2801761617

author_association: CONTRIBUTOR
user: login: scylladb-promoter
created_at: 2025-04-16T23:21:34Z
body: #### :green_circle: CI State: SUCCESS

:white_check_mark: - [Framework test](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16825/)
:white_check_mark: - [Build](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16825/)
:white_check_mark: - [Unit Tests Custom](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16825/)
**The following new/updated tests ran 100 times for each mode:**
:small_blue_diamond: cluster/object_store/test_backup
:white_check_mark: - [dtest with tablets](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-tablets/4822/)
:white_check_mark: - [dtest with consistent topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-consistent-topology-changes/7995/)
:white_check_mark: - [dtest with gossip topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release/16057/)
:white_check_mark: - [Unit Tests](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16825/)
#### Build Details:
- Duration: 8 hr 23 min
- Builder: i-0603afb4711ceff6d (m5d.16xlarge)

author_association: CONTRIBUTOR
user: login: xemul
created_at: 2025-04-17T09:47:34Z
body: LGTM, but inter-shard [ping-pong](https://github.com/scylladb/scylladb/pull/23727#discussion_r2048613818) to update progress is horrible
author_association: CONTRIBUTOR
user: login: avikivity
created_at: 2025-04-17T09:53:05Z
body: > > > > Please enhance the cover letter with an explanation of the algorithm and before/after test.
> > > 
> > > 
> > > Do you mean perf tests?
> > 
> > 
> > A test which will allow me or a user to say "this operation was improved by a factor of X". Could be `nodetool backup` or something.
> 
> please note [#23727 (comment)](https://github.com/scylladb/scylladb/pull/23727#issuecomment-2801761617)

We don't need a super duper full cluster performance test.

A patch that says "optimize" should be able to prove it's actually improving things.
author_association: MEMBER
user: login: kreuzerkrieg
created_at: 2025-04-17T11:22:11Z
body: > > > > > Please enhance the cover letter with an explanation of the algorithm and before/after test.
> > > > 
> > > > 
> > > > Do you mean perf tests?
> > > 
> > > 
> > > A test which will allow me or a user to say "this operation was improved by a factor of X". Could be `nodetool backup` or something.
> > 
> > 
> > please note [#23727 (comment)](https://github.com/scylladb/scylladb/pull/23727#issuecomment-2801761617)
> 
> We don't need a super duper full cluster performance test.
> 
> A patch that says "optimize" should be able to prove it's actually improving things.

will try to put together something quick

author_association: CONTRIBUTOR
user: login: kreuzerkrieg
created_at: 2025-04-17T13:30:41Z
body: > > > > > Please enhance the cover letter with an explanation of the algorithm and before/after test.
> > > > 
> > > > 
> > > > Do you mean perf tests?
> > > 
> > > 
> > > A test which will allow me or a user to say "this operation was improved by a factor of X". Could be `nodetool backup` or something.
> > 
> > 
> > please note [#23727 (comment)](https://github.com/scylladb/scylladb/pull/23727#issuecomment-2801761617)
> 
> We don't need a super duper full cluster performance test.
> 
> A patch that says "optimize" should be able to prove it's actually improving things.

Added to cover letter
author_association: CONTRIBUTOR
user: login: scylladb-promoter
created_at: 2025-04-17T21:43:36Z
body: #### :green_circle: CI State: SUCCESS

:white_check_mark: - [Framework test](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16854/)
:white_check_mark: - [Build](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16854/)
:white_check_mark: - [Unit Tests Custom](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16854/)
**The following new/updated tests ran 100 times for each mode:**
:small_blue_diamond: cluster/object_store/test_backup
:white_check_mark: - [dtest with tablets](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-tablets/4839/)
:white_check_mark: - [dtest with consistent topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release-with-consistent-topology-changes/8012/)
:white_check_mark: - [dtest with gossip topology changes](https://jenkins.scylladb.com/job/scylla-master/job/gating-dtest-release/16079/)
:white_check_mark: - [Unit Tests](https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/16854/)
#### Build Details:
- Duration: 8 hr 11 min
- Builder: spider4.cloudius-systems.com

author_association: CONTRIBUTOR