Repository: ceph/ceph
Pull request #62080: crimson/osd/pg_recovery: use OperationThrottler to throttle object pushes/pulls
Author: login: xxhdx1985126
Labels: name: common
name: crimson
name: wip-matan-testing
name: needs-crimson-qa
Created: 2025-03-03T02:44:21Z | Merged: 2025-04-07T08:12:31Z

@Matan-B @athanatos This PR reverts 14463aa7a493dc6903d18af960aac24e8ab8153b and use `OperationThrottler` to throttle recovery/backfill pushes/pulls instead of recovery/backfill operations.

Fixes: https://tracker.ceph.com/issues/70180
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>


<!--
  - Please give your pull request a title like

      [component]: [short description]

  - Please use this format for each git commit message:

      [component]: [short description]

      [A longer multiline description]

      Fixes: [ticket URL on tracker.ceph.com, create one if necessary]
      Signed-off-by: [Your Name] <[your email]>

    For examples, use "git log".
-->

## Contribution Guidelines
- To sign and title your commits, please refer to [Submitting Patches to Ceph](https://github.com/ceph/ceph/blob/main/SubmittingPatches.rst).

- If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to [Submitting Patches to Ceph - Backports](https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst) for the proper workflow.

- When filling out the below checklist, you may click boxes directly in the GitHub web UI.  When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an `x` between the brackets: `[x]`.  Spaces and capitalization matter when checking off items this way.

## Checklist
- Tracker (select at least one)
  - [x] References tracker ticket
  - [ ] Very recent bug; references commit where it was introduced
  - [ ] New feature (ticket optional)
  - [ ] Doc update (no ticket needed)
  - [ ] Code cleanup (no ticket needed)
- Component impact
  - [ ] Affects [Dashboard](https://tracker.ceph.com/projects/dashboard/issues/new), opened tracker ticket
  - [ ] Affects [Orchestrator](https://tracker.ceph.com/projects/orchestrator/issues/new), opened tracker ticket
  - [x] No impact that needs to be tracked
- Documentation (select at least one)
  - [ ] Updates relevant documentation
  - [x] No doc update is appropriate
- Tests (select at least one)
  - [ ] Includes [unit test(s)](https://docs.ceph.com/en/latest/dev/developer_guide/tests-unit-tests/)
  - [ ] Includes [integration test(s)](https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/)
  - [ ] Includes bug reproducer
  - [x] No tests

<details>
<summary>Show available Jenkins commands</summary>

- `jenkins test classic perf` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-perf-classic/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml)
- `jenkins test crimson perf` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-perf-crimson/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml)
- `jenkins test signed` [Jenkins Job](https://jenkins.ceph.com/job/ceph-pr-commits/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-pr-commits/config/definitions/ceph-pr-commits.yml)
- `jenkins test make check` [Jenkins Job](https://jenkins.ceph.com/job/ceph-pull-requests/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-pull-requests/config/definitions/ceph-pull-requests.yml)
- `jenkins test make check arm64` [Jenkins Job](https://jenkins.ceph.com/job/ceph-pull-requests-arm64/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-pull-requests-arm64/config/definitions/ceph-pull-requests-arm64.yml)
- `jenkins test submodules` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-pr-submodules/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-pr-submodules/config/definitions/ceph-pr-commits.yml)
- `jenkins test dashboard` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-dashboard-pull-requests/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-dashboard-pull-requests/config/definitions/ceph-dashboard-pull-requests.yml)
- `jenkins test dashboard cephadm` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-dashboard-cephadm-e2e/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-dashboard-cephadm-e2e/config/definitions/ceph-dashboard-cephadm-e2e.yml)
- `jenkins test api` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-api/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-pr-api/config/definitions/ceph-pr-api.yml)
- `jenkins test docs` [ReadTheDocs](https://readthedocs.org/projects/ceph/) | [Github Workflow Definition](https://github.com/ceph/ceph/blob/main/.readthedocs.yml)
- `jenkins test ceph-volume all` [Jenkins Jobs](https://jenkins.ceph.com/view/ceph-volume%20PR/) | [Jenkins Jobs Definition](https://github.com/ceph/ceph-build/blob/main/ceph-volume-cephadm-prs/config/definitions/ceph-volume-pr.yml)
- `jenkins test windows` [Jenkins Job](https://jenkins.ceph.com/job/ceph-windows-pull-requests/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-windows-pull-requests/config/definitions/ceph-windows-pull-requests.yml)
- `jenkins test rook e2e` [Jenkins Job](https://jenkins.ceph.com/view/all/job/ceph-orchestrator-rook-e2e/) | [Jenkins Job Definition](https://github.com/ceph/ceph-build/blob/main/ceph-rook-e2e/config/definitions/ceph-orchestrator-rook-e2e.yml)
</details>



Diff:
diff --git a/src/crimson/osd/backfill_state.cc b/src/crimson/osd/backfill_state.cc
index a068c1fa8d41b..7fea4b7d58423 100644
--- a/src/crimson/osd/backfill_state.cc
+++ b/src/crimson/osd/backfill_state.cc
@@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
       post_event(RequestWaiting{});
       return;
     } else if (should_rescan_replicas(backfill_state().peer_backfill_info,
diff --git a/src/crimson/osd/backfill_state.h b/src/crimson/osd/backfill_state.h
index ea4b159ded92f..75129d3974501 100644
--- a/src/crimson/osd/backfill_state.h
+++ b/src/crimson/osd/backfill_state.h
@@ -62,8 +62,6 @@ struct BackfillState {
   struct SuspendBackfill : sc::event<SuspendBackfill> {
   };
 
-  struct ThrottleAcquired : sc::event<ThrottleAcquired> {
-  };
 private:
   // internal events
   struct RequestPrimaryScanning : sc::event<RequestPrimaryScanning> {
@@ -262,7 +260,6 @@ struct BackfillState {
       sc::transition<RequestDone, Done>,
       sc::custom_reaction<SuspendBackfill>,
       sc::custom_reaction<Triggered>,
-      sc::transition<ThrottleAcquired, Enqueuing>,
       sc::transition<sc::event_base, Crashed>>;
     explicit Waiting(my_context);
     sc::result react(ObjectPushed);
diff --git a/src/crimson/osd/osd_operation.cc b/src/crimson/osd/osd_operation.cc
index 9780097097613..1416df4dac8a8 100644
--- a/src/crimson/osd/osd_operation.cc
+++ b/src/crimson/osd/osd_operation.cc
@@ -158,8 +158,7 @@ OperationThrottler::OperationThrottler(ConfigProxy &conf)
 
 void OperationThrottler::wake()
 {
-  while ((!max_in_progress || in_progress < max_in_progress) &&
-	 !scheduler->empty()) {
+  while (available() && !scheduler->empty()) {
     auto item = scheduler->dequeue();
     item.wake.set_value();
     ++in_progress;
diff --git a/src/crimson/osd/osd_operation.h b/src/crimson/osd/osd_operation.h
index d702919bd6fae..6e75f2f826f25 100644
--- a/src/crimson/osd/osd_operation.h
+++ b/src/crimson/osd/osd_operation.h
@@ -334,6 +334,22 @@ class OperationThrottler : public BlockerT<OperationThrottler>,
 			  const std::set<std::string> &changed) final;
   void update_from_config(const ConfigProxy &conf);
 
+  bool available() const {
+    return !max_in_progress || in_progress < max_in_progress;
+  }
+
+  template <typename F>
+  auto with_throttle(
+    crimson::osd::scheduler::params_t params,
+    F &&f) {
+    if (!max_in_progress) return f();
+    return acquire_throttle(params)
+      .then(std::forward<F>(f))
+      .finally([this] {
+	release_throttle();
+      });
+  }
+
   template <class OpT, class... Args>
   seastar::future<> with_throttle_while(
     BlockingEvent::Trigger<OpT>&& trigger,
@@ -342,18 +358,6 @@ class OperationThrottler : public BlockerT<OperationThrottler>,
       with_throttle_while(std::forward<Args>(args)...), *this);
   }
 
-  // Returns std::nullopt if the throttle is acquired immediately,
-  // returns the future for the acquiring otherwise
-  std::optional<seastar::future<>>
-  try_acquire_throttle_now(crimson::osd::scheduler::params_t params) {
-    if (!max_in_progress || in_progress < max_in_progress) {
-      ++in_progress;
-      --pending;
-      return std::nullopt;
-    }
-    return acquire_throttle(params);
-  }
-
 private:
   void dump_detail(Formatter *f) const final;
 
diff --git a/src/crimson/osd/osd_operations/background_recovery.cc b/src/crimson/osd/osd_operations/background_recovery.cc
index 7f93e42d53d8e..dbfeab07e6b69 100644
--- a/src/crimson/osd/osd_operations/background_recovery.cc
+++ b/src/crimson/osd/osd_operations/background_recovery.cc
@@ -77,18 +77,19 @@ seastar::future<> BackgroundRecoveryT<T>::start()
       std::chrono::milliseconds(std::lround(delay * 1000)));
   }
   return maybe_delay.then([ref, this] {
-    return this->template with_blocking_event<OperationThrottler::BlockingEvent>(
-      [ref, this] (auto&& trigger) {
-      return ss.with_throttle_while(
-        std::move(trigger),
-        this, get_scheduler_params(), [this] {
-          return interruptor::with_interruption([this] {
-            return do_recovery();
-          }, [](std::exception_ptr) {
-            return seastar::make_ready_future<bool>(false);
-          }, pg, epoch_started);
-        });
+    return seastar::repeat([ref, this] {
+      return interruptor::with_interruption([this] {
+       return do_recovery();
+      }, [](std::exception_ptr) {
+       return seastar::make_ready_future<bool>(false);
+      }, pg, epoch_started).then([](bool recovery_done) {
+       if (recovery_done) {
+         return seastar::stop_iteration::yes;
+       } else {
+         return seastar::stop_iteration::no;
+       }
       });
+    });
   });
 }
 
@@ -117,7 +118,8 @@ UrgentRecovery::do_recovery()
   ).then_interruptible([this] {
     return with_blocking_event<RecoveryBackend::RecoveryBlockingEvent,
 			       interruptor>([this] (auto&& trigger) {
-      return pg->get_recovery_handler()->recover_missing(trigger, soid, need);
+      return pg->get_recovery_handler()->recover_missing(
+	trigger, soid, need, false);
     }).then_interruptible([] {
       return seastar::make_ready_future<bool>(false);
     });
diff --git a/src/crimson/osd/osd_operations/background_recovery.h b/src/crimson/osd/osd_operations/background_recovery.h
index 255a934cd4987..2b3dbb121ae9f 100644
--- a/src/crimson/osd/osd_operations/background_recovery.h
+++ b/src/crimson/osd/osd_operations/background_recovery.h
@@ -66,7 +66,6 @@ class UrgentRecovery final : public BackgroundRecoveryT<UrgentRecovery> {
   void print(std::ostream&) const final;
 
   std::tuple<
-    OperationThrottler::BlockingEvent,
     RecoveryBackend::RecoveryBlockingEvent
   > tracking_events;
 
@@ -86,7 +85,6 @@ class PglogBasedRecovery final : public BackgroundRecoveryT<PglogBasedRecovery>
     float delay = 0);
 
   std::tuple<
-    OperationThrottler::BlockingEvent,
     RecoveryBackend::RecoveryBlockingEvent
   > tracking_events;
 
diff --git a/src/crimson/osd/pg_recovery.cc b/src/crimson/osd/pg_recovery.cc
index 735dd7f02e81f..f219411acda9d 100644
--- a/src/crimson/osd/pg_recovery.cc
+++ b/src/crimson/osd/pg_recovery.cc
@@ -67,6 +67,8 @@ PGRecovery::start_recovery_ops(
   if (max_to_start > 0) {
     max_to_start -= start_replica_recovery_ops(trigger, max_to_start, &started);
   }
+  using interruptor =
+    crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>;
   return interruptor::parallel_for_each(started,
 					[] (auto&& ifut) {
     return std::move(ifut);
@@ -108,7 +110,7 @@ PGRecovery::start_recovery_ops(
       }
       pg->reset_pglog_based_recovery_op();
     }
-    return seastar::make_ready_future<bool>(!done);
+    return seastar::make_ready_future<bool>(done);
   });
 }
 
@@ -194,10 +196,10 @@ size_t PGRecovery::start_primary_recovery_ops(
 	auto it = missing.get_items().find(head);
 	assert(it != missing.get_items().end());
 	auto head_need = it->second.need;
-	out->emplace_back(recover_missing(trigger, head, head_need));
+	out->emplace_back(recover_missing(trigger, head, head_need, true));
 	++skipped;
       } else {
-	out->emplace_back(recover_missing(trigger, soid, item.need));
+	out->emplace_back(recover_missing(trigger, soid, item.need, true));
       }
       ++started;
     }
@@ -304,7 +306,9 @@ size_t PGRecovery::start_replica_recovery_ops(
 PGRecovery::interruptible_future<>
 PGRecovery::recover_missing(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
-  const hobject_t &soid, eversion_t need)
+  const hobject_t &soid,
+  eversion_t need,
+  bool with_throttle)
 {
   logger().info("{} {} v {}", __func__, soid, need);
   auto [recovering, added] = pg->get_recovery_backend()->add_recovering(soid);
@@ -317,7 +321,9 @@ PGRecovery::recover_missing(
     } else {
       return recovering.wait_track_blocking(
 	trigger,
-	pg->get_recovery_backend()->recover_object(soid, need)
+	with_throttle
+	  ? recover_object_with_throttle(soid, need)
+	  : recover_object(soid, need)
 	.handle_exception_interruptible(
 	  [=, this, soid = std::move(soid)] (auto e) {
 	  on_failed_recover({ pg->get_pg_whoami() }, soid, need);
@@ -365,7 +371,7 @@ RecoveryBackend::interruptible_future<> PGRecovery::prep_object_replica_pushes(
     logger().info("{} {} v {}, new recovery", __func__, soid, need);
     return recovering.wait_track_blocking(
       trigger,
-      pg->get_recovery_backend()->recover_object(soid, need)
+      recover_object_with_throttle(soid, need)
       .handle_exception_interruptible(
 	[=, this, soid = std::move(soid)] (auto e) {
 	on_failed_recover({ pg->get_pg_whoami() }, soid, need);
@@ -514,6 +520,25 @@ void PGRecovery::request_primary_scan(
   });
 }
 
+PGRecovery::interruptible_future<>
+PGRecovery::recover_object_with_throttle(
+  const hobject_t &soid,
+  eversion_t need)
+{
+  crimson::osd::scheduler::params_t params =
+    {1, 0, crimson::osd::scheduler::scheduler_class_t::background_best_effort};
+  auto &ss = pg->get_shard_services();
+  logger().debug("{} {}", soid, need);
+  return ss.with_throttle(
+    std::move(params),
+    [this, soid, need] {
+    logger().debug("got throttle: {} {}", soid, need);
+    auto backend = pg->get_recovery_backend();
+    assert(backend);
+    return backend->recover_object(soid, need);
+  });
+}
+
 void PGRecovery::enqueue_push(
   const hobject_t& obj,
   const eversion_t& v,
@@ -525,7 +550,7 @@ void PGRecovery::enqueue_push(
   if (!added)
     return;
   peering_state.prepare_backfill_for_missing(obj, v, peers);
-  std::ignore = pg->get_recovery_backend()->recover_object(obj, v).\
+  std::ignore = recover_object_with_throttle(obj, v).\
   handle_exception_interruptible([] (auto) {
     ceph_abort_msg("got exception on backfill's push");
     return seastar::make_ready_future<>();
@@ -603,21 +628,8 @@ void PGRecovery::update_peers_last_backfill(
 
 bool PGRecovery::budget_available() const
 {
-  crimson::osd::scheduler::params_t params =
-    {1, 0, crimson::osd::scheduler::scheduler_class_t::background_best_effort};
   auto &ss = pg->get_shard_services();
-  auto futopt = ss.try_acquire_throttle_now(std::move(params));
-  if (!futopt) {
-    return true;
-  }
-  std::ignore = interruptor::make_interruptible(std::move(*futopt)
-  ).then_interruptible([this] {
-    assert(!backfill_state->is_triggered());
-    using BackfillState = crimson::osd::BackfillState;
-    backfill_state->process_event(
-      BackfillState::ThrottleAcquired{}.intrusive_from_this());
-  });
-  return false;
+  return ss.throttle_available();
 }
 
 void PGRecovery::on_pg_clean()
diff --git a/src/crimson/osd/pg_recovery.h b/src/crimson/osd/pg_recovery.h
index a9a48cf465ce8..37fae278fa09f 100644
--- a/src/crimson/osd/pg_recovery.h
+++ b/src/crimson/osd/pg_recovery.h
@@ -25,8 +25,6 @@ class PGBackend;
 
 class PGRecovery : public crimson::osd::BackfillState::BackfillListener {
 public:
-  using interruptor =
-    crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>;
   template <typename T = void>
   using interruptible_future = RecoveryBackend::interruptible_future<T>;
   PGRecovery(PGRecoveryListener* pg) : pg(pg) {}
@@ -67,7 +65,9 @@ class PGRecovery : public crimson::osd::BackfillState::BackfillListener {
   }
   RecoveryBackend::interruptible_future<> recover_missing(
     RecoveryBackend::RecoveryBlockingEvent::TriggerI&,
-    const hobject_t &soid, eversion_t need);
+    const hobject_t &soid,
+    eversion_t need,
+    bool with_throttle);
   RecoveryBackend::interruptible_future<> prep_object_replica_deletes(
     RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
     const hobject_t& soid,
@@ -99,6 +99,18 @@ class PGRecovery : public crimson::osd::BackfillState::BackfillListener {
   friend class ReplicatedRecoveryBackend;
   friend class crimson::osd::UrgentRecovery;
 
+  interruptible_future<> recover_object_with_throttle(
+    const hobject_t &soid,
+    eversion_t need);
+
+  interruptible_future<> recover_object(
+    const hobject_t &soid,
+    eversion_t need) {
+    auto backend = pg->get_recovery_backend();
+    assert(backend);
+    return backend->recover_object(soid, need);
+  }
+
   // backfill begin
   std::unique_ptr<crimson::osd::BackfillState> backfill_state;
   std::map<pg_shard_t,
diff --git a/src/crimson/osd/shard_services.h b/src/crimson/osd/shard_services.h
index 400976d0db954..38daf776b9925 100644
--- a/src/crimson/osd/shard_services.h
+++ b/src/crimson/osd/shard_services.h
@@ -593,8 +593,7 @@ class ShardServices : public OSDMapService {
   }
 
   FORWARD_TO_OSD_SINGLETON(get_pool_info)
-  FORWARD(with_throttle_while, with_throttle_while, local_state.throttler)
-  FORWARD(try_acquire_throttle_now, try_acquire_throttle_now, local_state.throttler)
+  FORWARD(with_throttle, with_throttle, local_state.throttler)
 
   FORWARD_TO_OSD_SINGLETON(build_incremental_map_msg)
   FORWARD_TO_OSD_SINGLETON(send_incremental_map)
@@ -618,6 +617,9 @@ class ShardServices : public OSDMapService {
     snap_dump_reservations,
     snap_reserver.dump)
 
+  bool throttle_available() const {
+    return local_state.throttler.available();
+  }
 
   auto local_update_priority(
     singleton_orderer_t &orderer,


Reviews:
user: login: athanatos
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T23:43:11Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T01:21:04Z
user: login: Matan-B
body: This seems to work well, I've left a few comments.

When the actual throttle usage (wip #62238) is introduced and will support all the other interfaces - we could decide if we would like to keep the `budget_available` limit added here as well. (cc @mohit84)
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T11:34:24Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T06:11:24Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T10:17:38Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T10:36:37Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T11:10:07Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T11:49:12Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T12:03:46Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T13:32:45Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T15:01:56Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T08:14:14Z
user: login: Matan-B
body: Following up to my comment above, seems that now `PGRecovery::recover_object` acquires the throttle and then internally uses `ReplicatedRecoveryBackend::recover_object`.

Since backfill enqueue uses `ReplicatedRecoveryBackend::recover_object` directly, can we add a wrapper that would be used by both recovery types prior to actually calling `ReplicatedRecoveryBackend::recover_object`?

Could be something like `recover_object_with_throttle` - as we would still need the non throttling recover object for UrgentRecovery probably.

What do you think?

CC @mohit84.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T12:43:06Z
user: login: Matan-B
body: As mentioned above, backfill uses `ReplicatedRecoveryBackend::recover_object` directly (without `PGRecovery::recover_missing`) - the changes here won't affect backfill.

The with_throttle param should be moved to the recovery backend to be used by both pglog and backfill.


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T12:31:33Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T13:00:43Z
user: login: Matan-B
body: > Um, I'm a bit confused. IIUC, backfill calls PGRecovery::enqueue_push which further calls ReplicatedRecoveryBackend::recover_object to do the pushes. So I changed PGRecovery::enqueue_push to call PGRecovery::recover_object_with_throttle so that backfill pushes can also be throttled. Am I missing something?


My mistake! You are right. Overall lgtm, looks like an elegant way to throttle both any recovery operation.
Only comments are cleanup and previous comment regarding naming.

 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T14:11:23Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T07:04:27Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T07:05:26Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:11:55Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T09:35:42Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T10:31:42Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:45:40Z
user: login: Matan-B
body: lgtm, only comment is regarding naming above - moving this one to be tested in the meanwhile.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:46:27Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:50:07Z
user: login: mohit84
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:50:23Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:52:09Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T12:55:05Z
user: login: mohit84
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T03:53:01Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T05:45:38Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T05:46:04Z
user: login: Matan-B
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T08:20:26Z
user: login: xxhdx1985126
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T08:41:41Z

Review comments:
diff_hunk: @@ -26,6 +26,8 @@ using std::map;
 using std::set;
 using PglogBasedRecovery = crimson::osd::PglogBasedRecovery;
 
+std::atomic<uint64_t> PGRecovery::ongoing_pushes = 0;
path: src/crimson/osd/pg_recovery.cc
user: login: athanatos
body: Why does this need to be atomic?
created_at: 2025-03-05T23:43:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,6 +26,8 @@ using std::map;
 using std::set;
 using PglogBasedRecovery = crimson::osd::PglogBasedRecovery;
 
+std::atomic<uint64_t> PGRecovery::ongoing_pushes = 0;
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: Sorry, that's a mistake, will correct it.
created_at: 2025-03-06T01:21:03Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: I think that it's confusing that we support both `osd_recovery_max_single_start` and `crimson_max_active_push_per_reactor`.

My suggestion would be to divide `osd_recovery_max_single_start` by `smp::count` to get the number of active pushes allowed for each reactor **instead** of introducing the new opiton.

```
      return pg->get_recovery_handler()->start_recovery_ops(
	trigger,
	*this,
	crimson::common::local_conf()->osd_recovery_max_single_start);
    });
```
```
PGRecovery::start_recovery_ops(
  RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
  PglogBasedRecovery &recover_op,
  size_t osd_max_to_start)
{
  ..
  auto max_active_pushes_per_reactor = osd_max_to_start / seastar::smp::count
  DEBUG(...)
```

What do you think?

created_at: 2025-03-12T11:22:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -527,10 +532,14 @@ void PGRecovery::enqueue_push(
   if (!added)
     return;
   peering_state.prepare_backfill_for_missing(obj, v, peers);
+  ongoing_pushes++;
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: We should probably log `ongoing_pushes` (increases/decreases) somewhere to keep track of it
created_at: 2025-03-12T11:25:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -605,8 +614,12 @@ void PGRecovery::update_peers_last_backfill(
 
 bool PGRecovery::budget_available() const
 {
-  // TODO: the limits!
-  return true;
+  auto max = crimson::common::get_conf<
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: Continuing my previous, I think we should be able to reuse `osd_recovery_max_single_start` here.
Let's also add a log line here with the three counters/limits to be aware that throttling has failed.

"max_active_push_per_reactor {}, max_active_push_per_osd {}, ongoing_pushes {}"
created_at: 2025-03-12T11:27:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -101,6 +101,7 @@ class PGRecovery : public crimson::osd::BackfillState::BackfillListener {
   std::unique_ptr<crimson::osd::BackfillState> backfill_state;
   std::map<pg_shard_t,
            MURef<MOSDPGBackfillRemove>> backfill_drop_requests;
+  static thread_local uint64_t ongoing_pushes;
path: src/crimson/osd/pg_recovery.h
user: login: Matan-B
body: This should work, ack.
created_at: 2025-03-12T11:28:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: yeah, will do.
created_at: 2025-03-13T06:11:24Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: @Matan-B Hi, matan, after a second thought, `max_active_pushes_per_reactor` limits the active pushes like `osd_recovery_max_active` does, so maybe `auto max_active_pushes_per_reactor = osd_recovery_max_active / seastar::smp::count` is better? What do you think?
created_at: 2025-03-31T10:17:38Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: Seems correct.
created_at: 2025-03-31T10:36:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: > Seems correct.

I'm also trying to refactor this PR, changing `OperationThrottle` to work at the `push/pull` level as opposed to the current way, limiting the active recoveries/pushes for both pglog-based recoveries and backfills in the same shard. What do you think?
created_at: 2025-03-31T11:10:06Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: It does make sense to introduce both `osd_recovery_max_single_start` and `osd_recovery_max_active` in one go as their logic is somewhat intertwined (`OSDService::_maybe_queue_recovery()`)

Is it a follow up to Mohit's https://github.com/ceph/ceph/pull/62238 or a different version of it? (cc @mohit84)
created_at: 2025-03-31T11:49:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: Um, I mean that, at present, the `OperationThrottler` is used to throttle the pglog-based recovery operation which can contain multiple object pushes/pulls, as shown in the code: https://github.com/ceph/ceph/blob/7861012264b7dc9062f08e37d6b32ab8c88fdae6/src/crimson/osd/osd_operations/background_recovery.cc#L80-L91

I intend to turn to use `OperationThrottler` to throttle the number of active pushes/pulls instead of recovery operations, so both pglog-based recoveries and backfills in the same shard can be throttled together. I think this is orthogonal to #62238 , which tries to throttle recovery/backfill operations, am I right?
created_at: 2025-03-31T12:03:46Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: @Matan-B I mean something like the following, in which all single object recoveries in the same shard are throttled together:
```
diff --git a/src/crimson/osd/osd_operation.cc b/src/crimson/osd/osd_operation.cc
index 97800970976..e552c3482b0 100644
--- a/src/crimson/osd/osd_operation.cc
+++ b/src/crimson/osd/osd_operation.cc
@@ -158,8 +158,7 @@ OperationThrottler::OperationThrottler(ConfigProxy &conf)
 
 void OperationThrottler::wake()
 {
-  while ((!max_in_progress || in_progress < max_in_progress) &&
-        !scheduler->empty()) {
+  while (avaiable() && !scheduler->empty()) {
     auto item = scheduler->dequeue();
     item.wake.set_value();
     ++in_progress;
diff --git a/src/crimson/osd/osd_operation.h b/src/crimson/osd/osd_operation.h
index af983eae4ce..d619b0753f5 100644
--- a/src/crimson/osd/osd_operation.h
+++ b/src/crimson/osd/osd_operation.h
@@ -334,6 +334,22 @@ public:
                          const std::set<std::string> &changed) final;
   void update_from_config(const ConfigProxy &conf);
 
+  bool available() const {
+    return !max_in_progress || in_progress < max_in_progress;
+  }
+
+  template <typename F>
+  auto with_throttle(
+    crimson::osd::scheduler::params_t params,
+    F &&f) {
+    if (!max_in_progress) return f();
+    return acquire_throttle(params)
+      .then(std::forward<F>(f))
+      .then([this] {
+       release_throttle();
+      });
+  }
+
   template <class OpT, class... Args>
   seastar::future<> with_throttle_while(
     BlockingEvent::Trigger<OpT>&& trigger,
diff --git a/src/crimson/osd/pg_recovery.cc b/src/crimson/osd/pg_recovery.cc
index 8d5841758e3..e40adc25031 100644
--- a/src/crimson/osd/pg_recovery.cc
+++ b/src/crimson/osd/pg_recovery.cc
@@ -319,7 +319,7 @@ PGRecovery::recover_missing(
     } else {
       return recovering.wait_track_blocking(
        trigger,
-       pg->get_recovery_backend()->recover_object(soid, need)
+       recover_object(soid, need)
        .handle_exception_interruptible(
          [=, this, soid = std::move(soid)] (auto e) {
          on_failed_recover({ pg->get_pg_whoami() }, soid, need);
@@ -367,7 +367,7 @@ RecoveryBackend::interruptible_future<> PGRecovery::prep_object_replica_pushes(
     logger().info("{} {} v {}, new recovery", __func__, soid, need);
     return recovering.wait_track_blocking(
       trigger,
-      pg->get_recovery_backend()->recover_object(soid, need)
+      recover_object(soid, need)
       .handle_exception_interruptible(
        [=, this, soid = std::move(soid)] (auto e) {
        on_failed_recover({ pg->get_pg_whoami() }, soid, need);
@@ -516,6 +516,23 @@ void PGRecovery::request_primary_scan(
   });
 }
 
+PGRecovery::interruptible_future<>
+PGRecovery::recover_object(
+  const hobject_t &soid,
+  eversion_t need)
+{
+  crimson::osd::scheduler::params_t params =
+    {1, 0, crimson::osd::scheduler::scheduler_class_t::background_best_effort};
+  auto &ss = pg->get_shard_services();
+  return ss.with_throttle(
+    std::move(params),
+    [this, soid, need] {
+    auto backend = pg->get_recovery_backend();
+    assert(backend);
+    return backend->recover_object(soid, need);
+  });
+}
+
 void PGRecovery::enqueue_push(
   const hobject_t& obj,
   const eversion_t& v,
@@ -527,7 +544,7 @@ void PGRecovery::enqueue_push(
   if (!added)
     return;
   peering_state.prepare_backfill_for_missing(obj, v, peers);
-  std::ignore = pg->get_recovery_backend()->recover_object(obj, v).\
+  std::ignore = recover_object(obj, v).\
   handle_exception_interruptible([] (auto) {
     ceph_abort_msg("got exception on backfill's push");
     return seastar::make_ready_future<>();
@@ -605,8 +622,8 @@ void PGRecovery::update_peers_last_backfill(
 
 bool PGRecovery::budget_available() const
 {
-  // TODO: the limits!
-  return true;
+  auto &ss = pg->get_shard_services();
+  return ss.available();
 }
 
 void PGRecovery::on_pg_clean()
diff --git a/src/crimson/osd/pg_recovery.h b/src/crimson/osd/pg_recovery.h
index 9d4a4874402..2b4a88bc5d9 100644
--- a/src/crimson/osd/pg_recovery.h
+++ b/src/crimson/osd/pg_recovery.h
@@ -97,6 +97,10 @@ private:
   friend class ReplicatedRecoveryBackend;
   friend class crimson::osd::UrgentRecovery;
 
+  interruptible_future<> recover_object(
+    const hobject_t &soid,
+    eversion_t need);
+
   // backfill begin
   std::unique_ptr<crimson::osd::BackfillState> backfill_state;
   std::map<pg_shard_t,
```

What do you think?
created_at: 2025-03-31T13:32:45Z
author_association: MEMBER
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: Matan-B
body: I agree that we should be able to throttle based on the number of active pushes/pulls instead of recovery operations. Although I'm not sure this is orthogonal to https://github.com/ceph/ceph/pull/62238 - based on the latest discussion there - it seems that OperationThrottler should be used to throttle pushed/pulls as well and not the based on the entire backfill interval operation. Right?
created_at: 2025-03-31T15:01:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,10 +42,13 @@ PGRecovery::interruptible_future<bool>
 PGRecovery::start_recovery_ops(
   RecoveryBackend::RecoveryBlockingEvent::TriggerI& trigger,
   PglogBasedRecovery &recover_op,
-  size_t max_to_start)
+  size_t max)
 {
   assert(pg->is_primary());
   assert(pg->is_peered());
+  auto max_to_start = std::max(
+    max,
+    crimson::common::get_conf<uint64_t>("crimson_max_active_push_per_reactor"));
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: I think so:-)
created_at: 2025-04-01T08:14:14Z
author_association: MEMBER
diff_hunk: @@ -77,18 +77,19 @@ seastar::future<> BackgroundRecoveryT<T>::start()
       std::chrono::milliseconds(std::lround(delay * 1000)));
   }
   return maybe_delay.then([ref, this] {
-    return this->template with_blocking_event<OperationThrottler::BlockingEvent>(
-      [ref, this] (auto&& trigger) {
-      return ss.with_throttle_while(
-        std::move(trigger),
-        this, get_scheduler_params(), [this] {
-          return interruptor::with_interruption([this] {
-            return do_recovery();
-          }, [](std::exception_ptr) {
-            return seastar::make_ready_future<bool>(false);
-          }, pg, epoch_started);
-        });
+    return seastar::repeat([ref, this] {
+      return interruptor::with_interruption([this] {
+       return do_recovery();
+      }, [](std::exception_ptr) {
+       return seastar::make_ready_future<bool>(false);
+      }, pg, epoch_started).then([](bool goon) {
+       if (goon) {
path: src/crimson/osd/osd_operations/background_recovery.cc
user: login: Matan-B
body: I would suggest changing start_recovery_ops to return `done` (instead of `!done`)
And change `goon` to `recovery_done`:
```
if (recovery_done) {
  return seastar::stop_iteration::yes
} else {
 ..
```

Otherwise, the reader must go to the definition of `do_recovery` to understand what `goon` is.
created_at: 2025-04-02T12:28:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,18 +77,19 @@ seastar::future<> BackgroundRecoveryT<T>::start()
       std::chrono::milliseconds(std::lround(delay * 1000)));
   }
   return maybe_delay.then([ref, this] {
-    return this->template with_blocking_event<OperationThrottler::BlockingEvent>(
-      [ref, this] (auto&& trigger) {
-      return ss.with_throttle_while(
-        std::move(trigger),
-        this, get_scheduler_params(), [this] {
-          return interruptor::with_interruption([this] {
-            return do_recovery();
-          }, [](std::exception_ptr) {
-            return seastar::make_ready_future<bool>(false);
-          }, pg, epoch_started);
-        });
+    return seastar::repeat([ref, this] {
+      return interruptor::with_interruption([this] {
+       return do_recovery();
+      }, [](std::exception_ptr) {
+       return seastar::make_ready_future<bool>(false);
+      }, pg, epoch_started).then([](bool goon) {
+       if (goon) {
path: src/crimson/osd/osd_operations/background_recovery.cc
user: login: xxhdx1985126
body: Yeah, will do.
created_at: 2025-04-02T13:00:43Z
author_association: MEMBER
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: Matan-B
body: Do we still need to check for budget_available here with the new approach? IIUC `budget_available` can be removed now as the `with_throttle` (from `recover_object_with_throttle`) will handle `available()`
created_at: 2025-04-02T13:54:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: mohit84
body: I am getting crash after merging this pull request with mclock pull request, it seems interrupt_cond is not valid during process recover_object via interruptible

(gdb) bt
#0  0x00007fbb73cabe0c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fbb73c5eb46 in raise () from /lib64/libc.so.6
#2  0x0000000001b6ee1a in reraise_fatal (signum=6) at /ceph-src/ceph/src/crimson/common/fatal_signal.cc:41
#3  FatalSignal::signal_entry (signum=6, info=<optimized out>) at /ceph-src/ceph/src/crimson/common/fatal_signal.cc:62
#4  <signal handler called>
#5  0x00007fbb73cabe0c in __pthread_kill_implementation () from /lib64/libc.so.6
#6  0x00007fbb73c5eb46 in raise () from /lib64/libc.so.6
#7  0x00007fbb73c48833 in abort () from /lib64/libc.so.6
#8  0x00000000011d74cf in ceph::__ceph_assert_fail (assertion=<optimized out>, file=<optimized out>,
    line=<optimized out>, func=<optimized out>) at /ceph-src/ceph/src/crimson/common/assert.cc:27
#9  0x00000000011d752a in ceph::__ceph_assert_fail (ctx=...) at /ceph-src/ceph/src/crimson/common/assert.cc:14
#10 0x00000000016fbb5e in crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<void> >::then_interruptible<ReplicatedRecoveryBackend::recover_object(const hobject_t&, eversion_t)::<lambda()> > (
    func=..., this=0x7fff4e9fcb90) at /ceph-src/ceph/src/crimson/common/interruptible_future.h:485
#11 ReplicatedRecoveryBackend::recover_object (this=<optimized out>, soid=..., need=...)
    at /ceph-src/ceph/src/crimson/osd/replicated_recovery_backend.cc:60
#12 0x00000000016ab375 in operator() (__closure=0x600000bac2d0) at /ceph-src/ceph/src/crimson/osd/pg_recovery.cc:536
#13 std::__invoke_impl<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<> >, PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&> (__f=...)
    at /opt/rh/gcc-toolset-13/root/usr/include/c++/13/bits/invoke.h:61
#14 std::__invoke<PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&> (__fn=...)
    at /opt/rh/gcc-toolset-13/root/usr/include/c++/13/bits/invoke.h:97
#15 std::invoke<PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&> (__fn=...)
    at /opt/rh/gcc-toolset-13/root/usr/include/c++/13/functional:114
#16 seastar::internal::future_invoke<PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&, seastar::internal::monostate> (v=..., func=...) at /ceph-src/ceph/src/seastar/include/seastar/core/future.hh:1150
#17 operator() (__closure=<optimized out>) at /ceph-src/ceph/src/seastar/include/seastar/core/future.hh:1466
--Type <RET> for more, q to quit, c to continue without paging--
#18 seastar::futurize<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<void> > >::satisfy_with_result_of<seastar::internal::promise_base_with_type<void>, seastar::future<>::then_impl_nrvo<PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<> > >(PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&&)::<lambda(pr_type&&, PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&, seastar::future<>::future_state&&)>::<lambda()> > (func=..., pr=...)
    at /ceph-src/ceph/src/crimson/common/interruptible_future.h:1664
#19 operator() (__closure=<optimized out>, state=..., func=..., pr=...)
    at /ceph-src/ceph/src/seastar/include/seastar/core/future.hh:1462
#20 seastar::continuation<seastar::internal::promise_base_with_type<void>, PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>, seastar::future<>::then_impl_nrvo<PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, seastar::future<> > >(PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&&)::<lambda(pr_type&&, PGRecovery::recover_object_with_throttle(const hobject_t&, eversion_t)::<lambda()>&, seastar::future<>::future_state&&)>, void>::run_and_dispose(void) (this=0x600000bac2a0) at /ceph-src/ceph/src/seastar/include/seastar/core/future.hh:728
#21 0x0000000002b57558 in seastar::reactor::run_tasks (this=this@entry=0x600000106000, tq=...)
    at /ceph-src/ceph/src/seastar/src/core/reactor.cc:2608
#22 0x0000000002b579fd in seastar::reactor::run_some_tasks (this=this@entry=0x600000106000)
    at /ceph-src/ceph/src/seastar/src/core/reactor.cc:3080
#23 0x0000000002b9140e in seastar::reactor::run_some_tasks (this=0x600000106000)
    at /ceph-src/ceph/src/seastar/include/seastar/core/circular_buffer_fixed_capacity.hh:213
#24 seastar::reactor::do_run (this=0x600000106000) at /ceph-src/ceph/src/seastar/src/core/reactor.cc:3248
#25 0x0000000002b92119 in seastar::reactor::run (this=<optimized out>)
    at /ceph-src/ceph/src/seastar/src/core/reactor.cc:3138
#26 0x0000000002aff736 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (
    this=this@entry=0x7fff4e9fd790, ac=ac@entry=10, av=av@entry=0x3c12f7c0, func=...)
    at /ceph-src/ceph/src/seastar/src/core/app-template.cc:276
--Type <RET> for more, q to quit, c to continue without paging--
#27 0x0000000002afffef in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (
    this=this@entry=0x7fff4e9fd790, ac=10, av=0x3c12f7c0, func=...)
    at /ceph-src/ceph/src/seastar/src/core/app-template.cc:167
#28 0x0000000001309f85 in main (argc=<optimized out>, argv=<optimized out>)
    at /opt/rh/gcc-toolset-13/root/usr/include/c++/13/bits/stl_vector.h:992

created_at: 2025-04-03T07:04:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: mohit84
body: This is the combined patch link https://github.com/ceph/ceph/compare/mclock_scheduler_backfill_recovery
created_at: 2025-04-03T07:05:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: Matan-B
body: Thanks for sharing this one @mohit84. Next steps could probably be to see if you can reproduce this without your additional branch - so we would know if the issue is in this PR or not.
If it's related to this PR, please share the reproducer.
created_at: 2025-04-03T08:11:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: mohit84
body: Please ignore the issue is in my patch.
created_at: 2025-04-03T09:35:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: xxhdx1985126
body: > Do we still need to check for budget_available here with the new approach? IIUC `budget_available` can be removed now as the `with_throttle` (from `recover_object_with_throttle`) will handle `available()`

If we remove `budget_available`, the backfill process may keep going to until all objects that are need to be backfilled are scanned. Although we can throttle the backfills of individual objects, all metadata of those objects are kept in OSD's memory. And since a PG may have many objects, this may lead to memory usage issue. What do you think?
created_at: 2025-04-03T10:31:42Z
author_association: MEMBER
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: Matan-B
body: @xxhdx1985126, That means that we check for budget_available prior to scanning but not actually taking the throttle when scanning, the throttle will be only taken for actual individual object pushes. Right?
I'm ok with that for now, in the future we'll need to also throttle scans. (Can be out-of-scope).
created_at: 2025-04-03T12:45:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: mohit84
body: I have also finished my testing i am ok with current approach.
created_at: 2025-04-03T12:50:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: Matan-B
body: @mohit84, thanks for verifying this! Should we keep #62238 open?
created_at: 2025-04-03T12:52:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: mohit84
body: I have closed the request. 
created_at: 2025-04-03T12:55:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -514,6 +520,23 @@ void PGRecovery::request_primary_scan(
   });
 }
 
+PGRecovery::interruptible_future<>
+PGRecovery::recover_object_with_throttle(
+  const hobject_t &soid,
+  eversion_t need)
+{
path: src/crimson/osd/pg_recovery.cc
user: login: mohit84
body: I think we can push a log here to capture the details about ongoing operation so that it would be easier to analyze logs in case of throttle vs non throttle. 
created_at: 2025-04-04T03:53:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -514,6 +520,23 @@ void PGRecovery::request_primary_scan(
   });
 }
 
+PGRecovery::interruptible_future<>
+PGRecovery::recover_object_with_throttle(
+  const hobject_t &soid,
+  eversion_t need)
+{
path: src/crimson/osd/pg_recovery.cc
user: login: xxhdx1985126
body: Yeah, will do.
created_at: 2025-04-04T05:45:38Z
author_association: MEMBER
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: xxhdx1985126
body: > @xxhdx1985126, That means that we check for budget_available prior to scanning but not actually taking the throttle when scanning, the throttle will be only taken for actual individual object pushes. Right?

Yes, that's right.
created_at: 2025-04-04T05:46:04Z
author_association: MEMBER
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: Matan-B
body: Merged - follow up tracker opened: https://tracker.ceph.com/issues/70808 (@xxhdx1985126)
created_at: 2025-04-07T08:20:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -342,7 +342,6 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx)
 
   do {
     if (!backfill_listener().budget_available()) {
-      DEBUGDPP("throttle failed, turning to Waiting", pg());
path: src/crimson/osd/backfill_state.cc
user: login: xxhdx1985126
body: > Merged - follow up tracker opened: https://tracker.ceph.com/issues/70808 (@xxhdx1985126)

Got it:-)
created_at: 2025-04-07T08:41:41Z
author_association: MEMBER

PR comments:
user: login: mohit84
created_at: 2025-03-04T15:39:05Z
body: @xxhdx1985126 I think here you are not using OperationThrottler so we can;t say it(backfill) is supporting throttle, IIUC you have implemented a threshold limit to push backfill ops. 
author_association: CONTRIBUTOR
user: login: athanatos
created_at: 2025-03-05T23:46:51Z
body: https://github.com/ceph/ceph/commit/14463aa7a493dc6903d18af960aac24e8ab8153b did have a problem in that there was no mechanism to release the throttle -- that would need to be fixed regardless.

This new mechanism doesn't actually perform the function that the throttle would.  Because the throttle is an OSD (actually, shard)-wide mechanism across IO types (client, background recovery, etc), it can delay backfill operations to preserve capacity for client IO.

We may, also, want to limit the number of objects we push concurrently, but we'd also need to use the throttle in order to prioritize backfill against other IO types.
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-03-06T01:22:22Z
body: > [14463aa](https://github.com/ceph/ceph/commit/14463aa7a493dc6903d18af960aac24e8ab8153b) did have a problem in that there was no mechanism to release the throttle -- that would need to be fixed regardless.
> 
> This new mechanism doesn't actually perform the function that the throttle would. Because the throttle is an OSD (actually, shard)-wide mechanism across IO types (client, background recovery, etc), it can delay backfill operations to preserve capacity for client IO.
> 
> We may, also, want to limit the number of objects we push concurrently, but we'd also need to use the throttle in order to prioritize backfill against other IO types.

So I think we need both mechinary, right? Does the backfill throttle has to be OSD-wide or reactor-wide is enough? Thansk:-)
author_association: MEMBER
user: login: athanatos
created_at: 2025-03-06T01:35:01Z
body: Reactor-wide should suffice.  For now, I think this is the right direction.  Let's revert the throttler change and introduce a reactor-wide push limit (should apply to log based recovery pushes as well, though that can be a follow up PR).  We can reintroduce the throttler later on with more appropriate interfaces.
author_association: CONTRIBUTOR
user: login: mohit84
created_at: 2025-03-06T10:44:12Z
body: @xxhdx1985126 Can you please refresh the pull request only revert the commit https://github.com/ceph/ceph/commit/14463aa7a493dc6903d18af960aac24e8ab8153b via tracker bug https://tracker.ceph.com/issues/70180, for push limit per pg specific change we should use a separate tracker bug.
author_association: CONTRIBUTOR
user: login: Matan-B
created_at: 2025-03-06T11:57:47Z
body: > @xxhdx1985126 Can you please refresh the pull request only revert the commit [14463aa](https://github.com/ceph/ceph/commit/14463aa7a493dc6903d18af960aac24e8ab8153b) via tracker bug https://tracker.ceph.com/issues/70180, for push limit per pg specific change we should use a separate tracker bug.

Let's keep this PR as is (with the revert and the new implementation).
If you need the revert earlier than the time this PR gets merged - feel free to push the revert as a separate PR so @xxhdx1985126 could rebase later.
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-03-12T02:15:58Z
body: @Matan-B @athanatos I've just modified the code as suggested, please take a look, again. Thanks:-)
author_association: MEMBER
user: login: mohit84
created_at: 2025-03-12T03:20:13Z
body: > osd_recovery_max_active

@Matan-B 

I might be wrong but i believe instead of using budget_available during Enqueuing we should use it on_backfill_reserved.
I am also planning to use throttle in the same function, i think if we call budget_available in the function backfill_reserved
it should be similar to classic OSD.
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-03-12T03:35:09Z
body: > > osd_recovery_max_active
> 
> @Matan-B
> 
> I might be wrong but i believe instead of using budget_available during Enqueuing we should use it on_backfill_reserved. I am also planning to use throttle in the same function, i think if we call budget_available in the function backfill_reserved it should be similar to classic OSD.

The purpose here is to throttle the maximum in-flight pushes, `on_backfill_reserved` happens only when the backfill is reserved at the pg level, it doesn't provide the information about the start/finish of the pushes, which, I think, makes it an inapprioprate place to do the throttling.
author_association: MEMBER
user: login: mohit84
created_at: 2025-03-12T03:45:42Z
body: > osd_recovery_max_active


If we do check the classic OSD source code how do we control active recovery ops? Before enqueue an object into scheduler(via function _maybe_queue_recovery) we do check the current active value if it is less than max value we do enqueue an object otherwise ignore it. I believe the behavior should be same in crimson also.
author_association: CONTRIBUTOR
user: login: mohit84
created_at: 2025-03-12T12:09:17Z
body: > This seems to work well, I've left a few comments.
> 
> When the actual throttle usage (wip #62238) is introduced and will support all the other interfaces - we could decide if we would like to keep the `budget_available` limit added here as well. (cc @mohit84)

Ack
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-04-01T09:18:46Z
body: @Matan-B I've just repushed the PR, please take a look, again. Thanks:-)
author_association: MEMBER
user: login: mohit84
created_at: 2025-04-01T09:54:23Z
body: > @Matan-B I've just repushed the PR, please take a look, again. Thanks:-)


I feel it is not a right place(during update_on_peers) to handle OperationThrottle for backfilling operation. Here we  are acquiring a throttler for specific sub operation recover_object not for complete backfill  operation . I think for backfilling also we need to do the same as we are doing for BackgroundRecovery. 
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-04-01T10:49:11Z
body: >The following tests FAILED:
	 32 - run-rbd-unit-tests-61.sh (Failed)
author_association: MEMBER
user: login: xxhdx1985126
created_at: 2025-04-01T10:49:17Z
body: jenkins test make check
author_association: MEMBER
user: login: Matan-B
created_at: 2025-04-01T12:36:09Z
body: > I feel it is not a right place(during update_on_peers) to handle OperationThrottle for backfilling operation. Here we are acquiring a throttler for specific sub operation recover_object not for complete backfill operation . I think for backfilling also we need to do the same as we are doing for BackgroundRecovery.

I still haven't reviewed the PR, but recover_object could be a good place to acquire the throttler as it's common to Backfiil/PGLog-based (with all it's variants: Replica/Primary/UrgentRecovery) recovery. That means that once we are properly throttling pushes/pulls in recover_object we control **any** recovery ongoing. The only downside is that we won't be able to differentiate between Backfill or pglog based recovery ops - though I'm ok with this as I like the generalization. @mohit84, what do you think?
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-04-01T12:45:26Z
body: > Following up to my comment above, seems that now `PGRecovery::recover_object` acquires the throttle and then internally uses `ReplicatedRecoveryBackend::recover_object`.
> 
> Since backfill enqueue uses `ReplicatedRecoveryBackend::recover_object` directly, can we add a wrapper that would be used by both recovery types prior to actually calling `ReplicatedRecoveryBackend::recover_object`?
> 
> Could be something like `recover_object_with_throttle` - as we would still need the non throttling recover object for UrgentRecovery probably.
> 
> What do you think?

Ah, yes, will do.

> 
> CC @mohit84.


author_association: MEMBER
user: login: mohit84
created_at: 2025-04-01T13:04:08Z
body: > recover_object

I have checked the code i think we can go with (recover_object) , this is more cleaner as compare to previous approach.
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-04-02T02:07:19Z
body: > Following up to my comment above, seems that now `PGRecovery::recover_object` acquires the throttle and then internally uses `ReplicatedRecoveryBackend::recover_object`.
> 
> Since backfill enqueue uses `ReplicatedRecoveryBackend::recover_object` directly, can we add a wrapper that would be used by both recovery types prior to actually calling `ReplicatedRecoveryBackend::recover_object`?
> 
> Could be something like `recover_object_with_throttle` - as we would still need the non throttling recover object for UrgentRecovery probably.
> 
> What do you think?

Done, please take a look, again. Thanks:-)


author_association: MEMBER
user: login: xxhdx1985126
created_at: 2025-04-02T13:00:04Z
body: > As mentioned above, backfill uses `ReplicatedRecoveryBackend::recover_object` directly (without `PGRecovery::recover_missing`) - the changes here won't affect backfill.

Um, I'm a bit confused. IIUC, backfill calls `PGRecovery::enqueue_push` which further calls `ReplicatedRecoveryBackend::recover_object` to do the pushes. So I changed `PGRecovery::enqueue_push` to call `PGRecovery::recover_object_with_throttle` so that backfill pushes can also be throttled. Am I missing something?
> 
> The with_throttle param should be moved to the recovery backend to be used by both pglog and backfill.


author_association: MEMBER
user: login: Matan-B
created_at: 2025-04-03T12:48:50Z
body: https://shaman.ceph.com/builds/ceph/wip-matanb-crimson-testing-62080
author_association: CONTRIBUTOR
user: login: xxhdx1985126
created_at: 2025-04-04T09:06:18Z
body: All done.
author_association: MEMBER
user: login: Matan-B
created_at: 2025-04-07T08:11:36Z
body: https://pulpito.ceph.com/matan-2025-04-06_13:17:00-crimson-rados-wip-matanb-crimson-testing-62080-distro-crimson-smithi/

```
8226288, 8226340 - https://tracker.ceph.com/issues/69612
8226384 - https://tracker.ceph.com/issues/67972
8226362 - https://tracker.ceph.com/issues/70502
8226405 - https://tracker.ceph.com/issues/69439
8226342, 8226378 -  https://tracker.ceph.com/issues/69406
```
author_association: CONTRIBUTOR