Repository: ceph/ceph
Pull request #61441: rgw: allow per-bucket minimum number of shards
Author: login: ivancich
Labels: name: feature
name: documentation
name: rgw
name: tests
name: needs-doc
name: TESTED
Created: 2025-01-17T22:31:52Z | Merged: 2025-03-01T15:41:01Z

Dynamic resharding can now reduce the number of shards. The code currently has a hard-coded value of 11 as the minimum number of shards dynamic resharding can reshard to. There may be cases where the user wants to set an alternate minimum, such as when they have a sense of how many objects the bucket will eventually hold.

This PR builds off of https://github.com/ceph/ceph/pull/61269 .

That PR allows the user to specify an initial number of shards during bucket creation. This PR then takes that number to be the minimum and saves it in the layout field of the bucket instance object (RGWBucketInfo).

When dynamic resharding is triggered, it will use that stored value as a minimum number of shards for resharing.

Fixes: https://tracker.ceph.com/issues/69464

<!--
  - Please give your pull request a title like

      [component]: [short description]

  - Please use this format for each git commit message:

      [component]: [short description]

      [A longer multiline description]

      Fixes: [ticket URL on tracker.ceph.com, create one if necessary]
      Signed-off-by: [Your Name] <[your email]>

    For examples, use "git log".
-->

## Contribution Guidelines
- To sign and title your commits, please refer to [Submitting Patches to Ceph](https://github.com/ceph/ceph/blob/main/SubmittingPatches.rst).

- If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to [Submitting Patches to Ceph - Backports](https://github.com/ceph/ceph/blob/master/SubmittingPatches-backports.rst) for the proper workflow.

- When filling out the below checklist, you may click boxes directly in the GitHub web UI.  When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an `x` between the brackets: `[x]`.  Spaces and capitalization matter when checking off items this way.

## Checklist
- Tracker (select at least one)
  - [x] References tracker ticket
  - [ ] Very recent bug; references commit where it was introduced
  - [ ] New feature (ticket optional)
  - [ ] Doc update (no ticket needed)
  - [ ] Code cleanup (no ticket needed)
- Component impact
  - [ ] Affects [Dashboard](https://tracker.ceph.com/projects/dashboard/issues/new), opened tracker ticket
  - [ ] Affects [Orchestrator](https://tracker.ceph.com/projects/orchestrator/issues/new), opened tracker ticket
  - [x] No impact that needs to be tracked
- Documentation (select at least one)
  - [x] Updates relevant documentation
  - [ ] No doc update is appropriate
- Tests (select at least one)
  - [ ] Includes [unit test(s)](https://docs.ceph.com/en/latest/dev/developer_guide/tests-unit-tests/)
  - [ ] Includes [integration test(s)](https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/)
  - [ ] Includes bug reproducer
  - [x] No tests

<details>
<summary>Show available Jenkins commands</summary>

- `jenkins retest this please`
- `jenkins test classic perf`
- `jenkins test crimson perf`
- `jenkins test signed`
- `jenkins test make check`
- `jenkins test make check arm64`
- `jenkins test submodules`
- `jenkins test dashboard`
- `jenkins test dashboard cephadm`
- `jenkins test api`
- `jenkins test docs`
- `jenkins render docs`
- `jenkins test ceph-volume all`
- `jenkins test ceph-volume tox`
- `jenkins test windows`
- `jenkins test rook e2e`
</details>



Diff:
diff --git a/doc/radosgw/dynamicresharding.rst b/doc/radosgw/dynamicresharding.rst
index b4e39388423b8..4c67c56c6f4ea 100644
--- a/doc/radosgw/dynamicresharding.rst
+++ b/doc/radosgw/dynamicresharding.rst
@@ -6,20 +6,20 @@ RGW Dynamic Bucket Index Resharding
 
 .. versionadded:: Luminous
 
-A large bucket index can lead to performance problems, which can
-be addressed by sharding bucket indexes.
-Until Luminous, changing the number of bucket shards (resharding)
-needed to be done offline, with RGW services disabled.
-Since the Luminous release Ceph has supported online bucket resharding.
+A bucket index object with too many entries can lead to performance
+problems. This can be addressed by resharding bucket indexes.  Until
+Luminous, changing the number of bucket shards (resharding) could only
+be done offline, with RGW services disabled.  Since the Luminous
+release Ceph has supported online bucket resharding.
 
 Each bucket index shard can handle its entries efficiently up until
-reaching a certain threshold. If this threshold is
-exceeded the system can suffer from performance issues. The dynamic
-resharding feature detects this situation and automatically increases
-the number of shards used by a bucket's index, resulting in a
-reduction of the number of entries in each shard. This
-process is transparent to the user. Writes to the target bucket
-are blocked (but reads are not) briefly during resharding process.
+reaching a certain threshold number. If this threshold is exceeded the
+system can suffer from performance issues. The dynamic resharding
+feature detects this situation and automatically increases the number
+of shards used by a bucket's index, resulting in a reduction of the
+number of entries in each shard. This process is transparent to the
+user. Writes to the target bucket can be blocked briefly during
+resharding process, but reads are not.
 
 By default dynamic bucket index resharding can only increase the
 number of bucket index shards to 1999, although this upper-bound is a
@@ -29,10 +29,16 @@ spread the number of entries across the bucket index
 shards more evenly.
 
 Detection of resharding opportunities runs as a background process
-that periodically
-scans all buckets. A bucket that requires resharding is added to
-a queue. A thread runs in the background and processes the queueued
-resharding tasks, one at a time and in order.
+that periodically scans all buckets. A bucket that requires resharding
+is added to a queue. A thread runs in the background and processes the
+queueued resharding tasks one at a time.
+
+Starting with Tentacle, dynamic resharding has the ability to reduce
+the number of shards. Once the condition allowing reduction is noted,
+there is a time delay before it will actually be executed, in case the
+number of objects increases in the near future. The goal of the delay
+is to avoid thrashing where resharding keeps getting re-invoked on
+buckets that fluctuate in numbers of objects.
 
 Multisite
 =========
@@ -48,6 +54,8 @@ Configuration
 .. confval:: rgw_dynamic_resharding
 .. confval:: rgw_max_objs_per_shard
 .. confval:: rgw_max_dynamic_shards
+.. confval:: rgw_dynamic_resharding_may_reduce
+.. confval:: rgw_dynamic_resharding_reduction_wait
 .. confval:: rgw_reshard_bucket_lock_duration
 .. confval:: rgw_reshard_thread_interval
 .. confval:: rgw_reshard_num_logs
@@ -136,7 +144,7 @@ For example, the output at each dynamic resharding stage is shown below:
 Cancel pending bucket resharding
 --------------------------------
 
-Note: Bucket resharding operations cannot be cancelled while executing. ::
+Note: Bucket resharding tasks cannot be cancelled once they start executing. ::
 
    # radosgw-admin reshard cancel --bucket <bucket_name>
 
@@ -158,6 +166,20 @@ since the former is prime. A variety of web sites have lists of prime
 numbers; search for "list of prime numbers" with your favorite
 search engine to locate some web sites.
 
+Setting a bucket's minimum number of shards
+-------------------------------------------
+
+::
+
+   # radosgw-admin bucket set-min-shards --bucket <bucket_name> --num-shards <min number of shards>
+
+Since dynamic resharding can now reduce the number of shards,
+administrators may want to prevent the number of shards from becoming
+too low, for example if the expect the number of objects to increase
+in the future. This command allows administrators to set a per-bucket
+minimum. This does not, however, prevent administrators from manually
+resharding to a lower number of shards.
+
 Troubleshooting
 ===============
 
diff --git a/src/rgw/driver/rados/rgw_bucket.cc b/src/rgw/driver/rados/rgw_bucket.cc
index e460739ac9f6f..e444521d395c9 100644
--- a/src/rgw/driver/rados/rgw_bucket.cc
+++ b/src/rgw/driver/rados/rgw_bucket.cc
@@ -2815,6 +2815,7 @@ void init_default_bucket_layout(CephContext *cct, rgw::BucketLayout& layout,
 
   if (shards) {
     layout.current_index.layout.normal.num_shards = *shards;
+    layout.current_index.layout.normal.min_num_shards = *shards;
   } else if (cct->_conf->rgw_override_bucket_index_max_shards > 0) {
     layout.current_index.layout.normal.num_shards =
       cct->_conf->rgw_override_bucket_index_max_shards;
diff --git a/src/rgw/driver/rados/rgw_rados.cc b/src/rgw/driver/rados/rgw_rados.cc
index 5eb2f9d0a075a..959e2f7d9943a 100644
--- a/src/rgw/driver/rados/rgw_rados.cc
+++ b/src/rgw/driver/rados/rgw_rados.cc
@@ -10823,6 +10823,7 @@ int RGWRados::cls_bucket_head_async(const DoutPrefixProvider *dpp, const RGWBuck
 void RGWRados::calculate_preferred_shards(const DoutPrefixProvider* dpp,
 					  const uint64_t num_objs,
 					  const uint32_t num_source_shards,
+					  const uint32_t min_layout_shards,
 					  bool& need_resharding,
 					  uint32_t* suggested_num_shards)
 {
@@ -10834,6 +10835,7 @@ void RGWRados::calculate_preferred_shards(const DoutPrefixProvider* dpp,
 
   RGWBucketReshard::calculate_preferred_shards(dpp,
 					       max_dynamic_shards,
+					       min_layout_shards,
 					       max_objs_per_shard,
 					       is_multisite,
 					       num_objs,
@@ -10867,8 +10869,11 @@ int RGWRados::check_bucket_shards(const RGWBucketInfo& bucket_info,
   uint32_t suggested_num_shards = 0;
   const uint32_t num_source_shards =
     rgw::current_num_shards(bucket_info.layout);
+  const uint32_t min_layout_shards =
+    rgw::current_min_layout_shards(bucket_info.layout);
 
-  calculate_preferred_shards(dpp, num_objs, num_source_shards,
+  calculate_preferred_shards(dpp, num_objs,
+			     num_source_shards, min_layout_shards,
 			     need_resharding, &suggested_num_shards);
   if (! need_resharding) {
     return 0;
diff --git a/src/rgw/driver/rados/rgw_rados.h b/src/rgw/driver/rados/rgw_rados.h
index 868e5c95eacb6..22bc71690d194 100644
--- a/src/rgw/driver/rados/rgw_rados.h
+++ b/src/rgw/driver/rados/rgw_rados.h
@@ -1630,6 +1630,7 @@ int restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx,
   void calculate_preferred_shards(const DoutPrefixProvider* dpp,
 				  const uint64_t num_objs,
 				  const uint32_t current_shard_count,
+				  const uint32_t min_layout_shards,
 				  bool& need_resharding,
 				  uint32_t* suggested_num_shard_count = nullptr);
 
diff --git a/src/rgw/driver/rados/rgw_reshard.cc b/src/rgw/driver/rados/rgw_reshard.cc
index 82080c4a06f27..429b209bbac27 100644
--- a/src/rgw/driver/rados/rgw_reshard.cc
+++ b/src/rgw/driver/rados/rgw_reshard.cc
@@ -69,23 +69,13 @@ const std::initializer_list<uint16_t> RGWBucketReshard::reshard_primes = {
 };
 
 
-uint32_t RGWBucketReshard::get_prime_shard_count(
-  uint32_t shard_count,
-  uint32_t max_dynamic_shards,
-  uint32_t min_dynamic_shards)
-{
+uint32_t RGWBucketReshard::nearest_prime(uint32_t shard_count)  {
   uint32_t prime_shard_count =
     get_prime_shards_greater_or_equal(shard_count);
 
   // if we cannot find a larger prime number, then just use what was
   // passed in
-  if (! prime_shard_count) {
-    prime_shard_count = shard_count;
-  }
-
-  // keep within min/max bounds
-  return std::min(max_dynamic_shards,
-		  std::max(min_dynamic_shards, prime_shard_count));
+  return prime_shard_count ? prime_shard_count : shard_count;
 }
 
 
@@ -96,6 +86,7 @@ uint32_t RGWBucketReshard::get_prime_shard_count(
 void RGWBucketReshard::calculate_preferred_shards(
   const DoutPrefixProvider* dpp,
   const uint32_t max_dynamic_shards,
+  const uint32_t min_layout_shards,
   const uint64_t max_objs_per_shard,
   const bool is_multisite,
   const uint64_t num_objs,
@@ -139,10 +130,13 @@ void RGWBucketReshard::calculate_preferred_shards(
   }
 
   if (prefer_prime) {
-    calculated_num_shards = get_prime_shard_count(
-      calculated_num_shards, max_dynamic_shards, min_dynamic_shards);
+    calculated_num_shards = nearest_prime(calculated_num_shards);
   }
 
+  calculated_num_shards =
+    std::min(max_dynamic_shards,
+	     std::max({ calculated_num_shards, min_dynamic_shards, min_layout_shards }));
+
   ldpp_dout(dpp, 20) << __func__ << ": reshard " << verb <<
     " suggested; current average (objects/shard) is " <<
     float(num_objs) / current_num_shards << ", which is not within " <<
@@ -461,6 +455,7 @@ static int init_target_layout(rgw::sal::RadosStore* store,
   rgw::bucket_index_layout_generation target;
   target.layout.type = rgw::BucketIndexType::Normal;
   target.layout.normal.num_shards = new_num_shards;
+  target.layout.normal.min_num_shards = current.layout.normal.min_num_shards;
   target.gen = current.gen + 1;
 
   if (bucket_info.reshard_status == cls_rgw_reshard_status::IN_PROGRESS) {
@@ -1256,7 +1251,7 @@ int RGWBucketReshard::do_reshard(const rgw::bucket_index_layout_generation& curr
     // block the client op and complete the resharding
     ceph_assert(bucket_info.layout.resharding == rgw::BucketReshardState::InProgress);
     ret = reshard_process(current, max_op_entries, target_shards_mgr, verbose_json_out, out,
-                              formatter, bucket_info.layout.resharding, dpp, y);
+			  formatter, bucket_info.layout.resharding, dpp, y);
     if (ret < 0) {
       ldpp_dout(dpp, 0) << __func__ << ": failed in progress state of reshard ret = " << ret << dendl;
       return ret;
@@ -1637,6 +1632,9 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
     ret = store->getRados()->get_bucket_stats(dpp, bucket_info,
 					      bucket_info.layout.current_index,
 					      -1, nullptr, nullptr, stats, nullptr, nullptr);
+    if (ret < 0) {
+      return clean_up("unable to access buckets current stats");
+    }
 
     // determine current number of bucket entries across shards
     uint64_t num_entries = 0;
@@ -1645,7 +1643,9 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
     }
 
     const uint32_t current_shard_count =
-      rgw::num_shards(bucket_info.get_current_index().layout.normal);
+      rgw::current_num_shards(bucket_info.layout);
+    const uint32_t min_layout_shards =
+      rgw::current_min_layout_shards(bucket_info.layout);
 
     bool needs_resharding { false };
     uint32_t suggested_shard_count { 0 };
@@ -1653,7 +1653,7 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
     // needed to perform the calculation before calling
     // calculating_preferred_shards() in this class
     store->getRados()->calculate_preferred_shards(
-      dpp, num_entries, current_shard_count,
+      dpp, num_entries, current_shard_count, min_layout_shards,
       needs_resharding, &suggested_shard_count);
 
     // if we no longer need resharding or currently need to expand
@@ -1711,7 +1711,6 @@ int RGWReshard::process_entry(const cls_rgw_reshard_entry& entry,
   }
 
   // all checkes passed; we can reshard...
-
   RGWBucketReshard br(store, bucket_info, bucket_attrs, nullptr);
 
   ReshardFaultInjector f; // no fault injected
diff --git a/src/rgw/driver/rados/rgw_reshard.h b/src/rgw/driver/rados/rgw_reshard.h
index 3d056e50f468d..bdd7f67873540 100644
--- a/src/rgw/driver/rados/rgw_reshard.h
+++ b/src/rgw/driver/rados/rgw_reshard.h
@@ -160,14 +160,12 @@ class RGWBucketReshard {
     }
   }
 
-  // returns a preferred number of shards given a calculated number of
-  // shards based on max_dynamic_shards and the list of prime values
-  static uint32_t get_prime_shard_count(uint32_t suggested_shards,
-					uint32_t max_dynamic_shards,
-					uint32_t min_dynamic_shards);
+  // returns a preferred number of shards as a prime value
+  static uint32_t nearest_prime(uint32_t suggested_shards);
 
   static void calculate_preferred_shards(const DoutPrefixProvider* dpp,
 					 const uint32_t max_dynamic_shards,
+					 const uint32_t min_layout_shards,
 					 const uint64_t max_objs_per_shard,
 					 const bool is_multisite,
 					 const uint64_t num_objs,
diff --git a/src/rgw/radosgw-admin/radosgw-admin.cc b/src/rgw/radosgw-admin/radosgw-admin.cc
index 991f9779f0d06..6a53e77a7be58 100644
--- a/src/rgw/radosgw-admin/radosgw-admin.cc
+++ b/src/rgw/radosgw-admin/radosgw-admin.cc
@@ -1,10 +1,9 @@
-
 // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
 // vim: ts=8 sw=2 smarttab ft=cpp
 
 /*
  * Copyright (C) 2025 IBM 
-*/
+ */
 
 #include <cerrno>
 #include <string>
@@ -169,6 +168,7 @@ void usage()
   cout << "  bucket check unlinked            check for object versions that are not visible in a bucket listing \n";
   cout << "  bucket chown                     link bucket to specified user and update its object ACLs\n";
   cout << "  bucket reshard                   reshard bucket\n";
+  cout << "  bucket set-min-shards            set the minimum number of shards that dynamic resharding will consider for a bucket\n";
   cout << "  bucket rewrite                   rewrite all objects in the specified bucket\n";
   cout << "  bucket sync checkpoint           poll a bucket's sync status until it catches up to its remote\n";
   cout << "  bucket sync disable              disable bucket sync\n";
@@ -699,6 +699,7 @@ enum class OPT {
   BUCKET_RM,
   BUCKET_REWRITE,
   BUCKET_RESHARD,
+  BUCKET_SET_MIN_SHARDS,
   BUCKET_CHOWN,
   BUCKET_RADOS_LIST,
   BUCKET_SHARD_OBJECTS,
@@ -937,6 +938,7 @@ static SimpleCmd::Commands all_cmds = {
   { "bucket rm", OPT::BUCKET_RM },
   { "bucket rewrite", OPT::BUCKET_REWRITE },
   { "bucket reshard", OPT::BUCKET_RESHARD },
+  { "bucket set-min-shards", OPT::BUCKET_SET_MIN_SHARDS },
   { "bucket chown", OPT::BUCKET_CHOWN },
   { "bucket radoslist", OPT::BUCKET_RADOS_LIST },
   { "bucket rados list", OPT::BUCKET_RADOS_LIST },
@@ -8803,6 +8805,51 @@ int main(int argc, const char **argv)
     }
   } // OPT_RESHARD_CANCEL
 
+  if (opt_cmd == OPT::BUCKET_SET_MIN_SHARDS) {
+    if (bucket_name.empty()) {
+      cerr << "ERROR: bucket not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (!num_shards_specified) {
+      cerr << "ERROR: --num-shards not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (num_shards < 1) {
+      cerr << "ERROR: --num-shards must be at least 1" << std::endl;
+      return -EINVAL;
+    }
+
+    int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
+    if (ret < 0) {
+      return -ret;
+    }
+    auto& bucket_info = bucket->get_info();
+
+    const rgw::BucketIndexType type =
+      bucket_info.layout.current_index.layout.type;
+    if (type != rgw::BucketIndexType::Normal) {
+      cerr << "ERROR: the bucket's layout is type " << type <<
+	" instead of type " << rgw::BucketIndexType::Normal <<
+	" and therefore does not have a "
+	"minimum number of shards that can be altered" << std::endl;
+      return EINVAL;
+    }
+
+    uint32_t& min_num_shards =
+      bucket_info.layout.current_index.layout.normal.min_num_shards;
+    min_num_shards = num_shards;
+
+    ret = bucket->put_info(dpp(), false, real_time(), null_yield);
+    if (ret < 0) {
+      cerr << "ERROR: failed writing bucket instance info: " << cpp_strerror(-ret) << std::endl;
+      return -ret;
+    }
+
+    return 0;
+  } // SET_MIN_SHARDS
+
   if (opt_cmd == OPT::OBJECT_UNLINK) {
     int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
     if (ret < 0) {
diff --git a/src/rgw/rgw_bucket_layout.cc b/src/rgw/rgw_bucket_layout.cc
index 1f8db396a0d13..10ff7200eaa30 100644
--- a/src/rgw/rgw_bucket_layout.cc
+++ b/src/rgw/rgw_bucket_layout.cc
@@ -81,16 +81,20 @@ void decode_json_obj(BucketHashType& t, JSONObj *obj)
 // bucket_index_normal_layout
 void encode(const bucket_index_normal_layout& l, bufferlist& bl, uint64_t f)
 {
-  ENCODE_START(1, 1, bl);
+  ENCODE_START(2, 1, bl);
   encode(l.num_shards, bl);
   encode(l.hash_type, bl);
+  encode(l.min_num_shards, bl);
   ENCODE_FINISH(bl);
 }
 void decode(bucket_index_normal_layout& l, bufferlist::const_iterator& bl)
 {
-  DECODE_START(1, bl);
+  DECODE_START(2, bl);
   decode(l.num_shards, bl);
   decode(l.hash_type, bl);
+  if (struct_v >= 2) {
+    decode(l.min_num_shards, bl);
+  }
   DECODE_FINISH(bl);
 }
 void encode_json_impl(const char *name, const bucket_index_normal_layout& l, ceph::Formatter *f)
@@ -98,12 +102,16 @@ void encode_json_impl(const char *name, const bucket_index_normal_layout& l, cep
   f->open_object_section(name);
   encode_json("num_shards", l.num_shards, f);
   encode_json("hash_type", l.hash_type, f);
+  encode_json("min_num_shards", l.min_num_shards, f);
   f->close_section();
 }
 void decode_json_obj(bucket_index_normal_layout& l, JSONObj *obj)
 {
   JSONDecoder::decode_json("num_shards", l.num_shards, obj);
   JSONDecoder::decode_json("hash_type", l.hash_type, obj);
+
+  // if not set in json, set to default value of 1
+  JSONDecoder::decode_json("min_num_shards", l.min_num_shards, obj, 1);
 }
 
 // bucket_index_layout
diff --git a/src/rgw/rgw_bucket_layout.h b/src/rgw/rgw_bucket_layout.h
index b360dd32c3715..08bacc81b306b 100644
--- a/src/rgw/rgw_bucket_layout.h
+++ b/src/rgw/rgw_bucket_layout.h
@@ -54,10 +54,15 @@ void decode_json_obj(BucketHashType& t, JSONObj *obj);
 struct bucket_index_normal_layout {
   uint32_t num_shards = 1;
 
+  // the fewest number of shards this bucket layout allows
+  uint32_t min_num_shards = 1;
+
   BucketHashType hash_type = BucketHashType::Mod;
 
-  friend std::ostream& operator<<(std::ostream& out, const bucket_index_normal_layout& l) {
-    out << "num_shards=" << l.num_shards << ", hash_type=" << to_string(l.hash_type);
+  friend std::ostream& operator<<(std::ostream& out,
+				  const bucket_index_normal_layout& l) {
+    out << "num_shards=" << l.num_shards << ", min_num_shards=" <<
+      l.min_num_shards << ", hash_type=" << to_string(l.hash_type);
     return out;
   }
 };
@@ -278,9 +283,13 @@ inline uint32_t num_shards(const bucket_index_layout& index) {
 inline uint32_t num_shards(const bucket_index_layout_generation& index) {
   return num_shards(index.layout);
 }
+
 inline uint32_t current_num_shards(const BucketLayout& layout) {
   return num_shards(layout.current_index);
 }
+inline uint32_t current_min_layout_shards(const BucketLayout& layout) {
+  return layout.current_index.layout.normal.min_num_shards;
+}
 inline bool is_layout_indexless(const bucket_index_layout_generation& layout) {
   return layout.layout.type == BucketIndexType::Indexless;
 }
diff --git a/src/rgw/rgw_common.h b/src/rgw/rgw_common.h
index 51e4f385ea4be..73f4923b84036 100644
--- a/src/rgw/rgw_common.h
+++ b/src/rgw/rgw_common.h
@@ -1045,6 +1045,9 @@ class RGWSI_Zone;
 
 #include "rgw_cksum.h"
 
+
+// this represents the at-rest bucket instance object and is stored as
+// a system object
 struct RGWBucketInfo {
   rgw_bucket bucket;
   rgw_owner owner;
diff --git a/src/rgw/rgw_sal.h b/src/rgw/rgw_sal.h
index a4c1467f322fe..91d6aa6f77be5 100644
--- a/src/rgw/rgw_sal.h
+++ b/src/rgw/rgw_sal.h
@@ -686,7 +686,7 @@ class Driver {
 
     /** Register admin APIs unique to this driver */
     virtual void register_admin_apis(RGWRESTMgr* mgr) = 0;
-};
+}; // class Driver
 
 
 /// \brief Ref-counted callback object for User/Bucket read_stats_async().
diff --git a/src/test/cli/radosgw-admin/help.t b/src/test/cli/radosgw-admin/help.t
index c1675d11a80dc..a5559f8fe2d37 100644
--- a/src/test/cli/radosgw-admin/help.t
+++ b/src/test/cli/radosgw-admin/help.t
@@ -38,6 +38,7 @@
     bucket check unlinked            check for object versions that are not visible in a bucket listing 
     bucket chown                     link bucket to specified user and update its object ACLs
     bucket reshard                   reshard bucket
+    bucket set-min-shards            set the minimum number of shards that dynamic resharding will consider for a bucket
     bucket rewrite                   rewrite all objects in the specified bucket
     bucket sync checkpoint           poll a bucket's sync status until it catches up to its remote
     bucket sync disable              disable bucket sync
diff --git a/src/test/rgw/test_rgw_reshard.cc b/src/test/rgw/test_rgw_reshard.cc
index 3513e644aa88d..d014fb33aa1a6 100644
--- a/src/test/rgw/test_rgw_reshard.cc
+++ b/src/test/rgw/test_rgw_reshard.cc
@@ -16,12 +16,15 @@
 #include <gtest/gtest.h>
 
 
-TEST(TestRGWReshard, dynamic_reshard_shard_count)
+TEST(TestRGWReshard, max_prime_shards)
 {
   // assuming we have prime numbers up to 1999
   ASSERT_EQ(1999u, RGWBucketReshard::get_max_prime_shards()) <<
     "initial list has primes up to 1999";
+}
 
+TEST(TestRGWReshard, prime_lookups)
+{
   ASSERT_EQ(1u, RGWBucketReshard::get_prime_shards_greater_or_equal(1)) <<
     "we allow for 1 shard even though it's not prime";
   ASSERT_EQ(809u, RGWBucketReshard::get_prime_shards_greater_or_equal(808)) <<
@@ -47,24 +50,72 @@ TEST(TestRGWReshard, dynamic_reshard_shard_count)
     "811 is prime";
   ASSERT_EQ(811u, RGWBucketReshard::get_prime_shards_less_or_equal(812)) <<
     "821 is prime";
+}
 
+TEST(TestRGWReshard, nearest_prime)
+{
   // tests when max dynamic shards is equal to end of prime list
-  ASSERT_EQ(1999u, RGWBucketReshard::get_prime_shard_count(1998, 1999, 11));
-  ASSERT_EQ(1999u, RGWBucketReshard::get_prime_shard_count(1999, 1999, 11));
-  ASSERT_EQ(1999u, RGWBucketReshard::get_prime_shard_count(2000, 1999, 11));
-
-  // tests when max dynamic shards is above end of prime list
-  ASSERT_EQ(1999u, RGWBucketReshard::get_prime_shard_count(1998, 3000, 11));
-  ASSERT_EQ(1999u, RGWBucketReshard::get_prime_shard_count(1999, 3000, 11));
-  ASSERT_EQ(2000u, RGWBucketReshard::get_prime_shard_count(2000, 3000, 11));
-  ASSERT_EQ(2001u, RGWBucketReshard::get_prime_shard_count(2001, 3000, 11));
-
-  // tests when max dynamic shards is below end of prime list
-  ASSERT_EQ(500u, RGWBucketReshard::get_prime_shard_count(1998, 500, 11));
-  ASSERT_EQ(500u, RGWBucketReshard::get_prime_shard_count(2001, 500, 11));
-
-  // tests when max dynamic shards is below end of prime list
-  ASSERT_EQ(499u, RGWBucketReshard::get_prime_shard_count(498, 1999, 499));
-  ASSERT_EQ(499u, RGWBucketReshard::get_prime_shard_count(499, 1999, 499));
-  ASSERT_EQ(503u, RGWBucketReshard::get_prime_shard_count(500, 1999, 499));
+
+  ASSERT_EQ(239u, RGWBucketReshard::nearest_prime(238));
+  ASSERT_EQ(239u, RGWBucketReshard::nearest_prime(239));
+  ASSERT_EQ(241u, RGWBucketReshard::nearest_prime(240));
+  ASSERT_EQ(241u, RGWBucketReshard::nearest_prime(241));
+  ASSERT_EQ(251u, RGWBucketReshard::nearest_prime(242));
+
+  ASSERT_EQ(1997u, RGWBucketReshard::nearest_prime(1995));
+  ASSERT_EQ(1997u, RGWBucketReshard::nearest_prime(1996));
+  ASSERT_EQ(1997u, RGWBucketReshard::nearest_prime(1997));
+  ASSERT_EQ(1999u, RGWBucketReshard::nearest_prime(1998));
+  ASSERT_EQ(1999u, RGWBucketReshard::nearest_prime(1999));
+  ASSERT_EQ(2000u, RGWBucketReshard::nearest_prime(2000));
+}
+
+TEST(TestRGWReshard, calculate_preferred_shards)
+{
+  bool needs_resharding;
+  uint32_t suggested_shard_count = 0;
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 101, 100000, false, 10000000, 200,
+					       needs_resharding, &suggested_shard_count);
+
+  ASSERT_EQ(false, needs_resharding) << "no need to reshard when shards are half-used";
+
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 101, 100000, false, 20200000, 200,
+					       needs_resharding, &suggested_shard_count, false);
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(404, suggested_shard_count) << "number of shards when primes are not preferred";
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 101, 100000, false, 20200000, 200,
+					       needs_resharding, &suggested_shard_count, true);
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(409, suggested_shard_count) << "number of shards when primes are preferred";
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 101, 100000, true, 20200000, 200,
+					       needs_resharding, &suggested_shard_count, true);
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(1619, suggested_shard_count) <<
+    "number of shards under multisite with primes preferred since "
+    "multisite quadruples number of shards to reduce need to reshaard";
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 3, 100000, false, 650000, 700,
+					       needs_resharding, &suggested_shard_count, true);
+  // 650,000 objs across 700 shards -> <1000 objs per shard; 650000 /
+  // 50000 = 13
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(13, suggested_shard_count) << "shard reduction without hitting min_layout_shards";
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 3, 100000, false, 350000, 400,
+					       needs_resharding, &suggested_shard_count, true);
+  // 350,000 objs across 400 shards -> <1000 objs per shard; 350000 /
+  // 50000 = 7, but hard-coded minimum of 11
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(11, suggested_shard_count) << "shard reduction and hitting hard-coded minimum of 11";
+
+  RGWBucketReshard::calculate_preferred_shards(nullptr, 1999, 51, 100000, false, 650000, 700,
+					       needs_resharding, &suggested_shard_count, true);
+  // 650,000 objs across 700 shards -> <1000 objs per shard; 650000 /
+  // 50000 = 13, but bucket min of 51
+  ASSERT_EQ(true, needs_resharding);
+  ASSERT_EQ(51, suggested_shard_count) << "shard reduction and hitting min_layout_shards";
 }


Reviews:
user: login: cbodley
body: looks good. an admin command like `radosgw-admin bucket set-min-shards` could be useful to support changes to this value
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-01-22T15:18:48Z
user: login: cbodley
body: looks great! https://docs.ceph.com/en/latest/radosgw/dynamicresharding/ could use a blurb about the admin command and downsharding in general
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T18:52:13Z
user: login: cbodley
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T18:49:55Z
user: login: ivancich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T19:35:05Z
user: login: anthonyeleven
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T20:01:33Z
user: login: cbodley
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T20:19:08Z
user: login: ivancich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-01T00:29:42Z
user: login: ivancich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-01T00:33:37Z

Review comments:
diff_hunk: @@ -1521,10 +1521,20 @@ static int bucket_stats(rgw::sal::Driver* driver,
   ::encode_json("explicit_placement", bucket->get_key().explicit_placement, formatter);
   formatter->dump_string("id", bucket->get_bucket_id());
   formatter->dump_string("marker", bucket->get_marker());
-  formatter->dump_stream("index_type") << bucket_info.layout.current_index.layout.type;
   formatter->dump_int("index_generation", bucket_info.layout.current_index.gen);
-  formatter->dump_int("num_shards",
-		      bucket_info.layout.current_index.layout.normal.num_shards);
path: src/rgw/driver/rados/rgw_bucket.cc
user: login: cbodley
body: would prefer not to remove these fields in case users are parsing the output of `radosgw-admin bucket stats`

`min_num_shards` will already be dumped as part of `radosgw-admin bucket layout`, so i'm not sure we need to add a `current_layout` section here
created_at: 2025-01-22T15:16:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -81,29 +81,39 @@ void decode_json_obj(BucketHashType& t, JSONObj *obj)
 // bucket_index_normal_layout
 void encode(const bucket_index_normal_layout& l, bufferlist& bl, uint64_t f)
 {
-  ENCODE_START(1, 1, bl);
+  ENCODE_START(2, 1, bl);
   encode(l.num_shards, bl);
   encode(l.hash_type, bl);
+  encode(l.min_num_shards, bl);
   ENCODE_FINISH(bl);
 }
 void decode(bucket_index_normal_layout& l, bufferlist::const_iterator& bl)
 {
-  DECODE_START(1, bl);
+  DECODE_START(2, bl);
   decode(l.num_shards, bl);
   decode(l.hash_type, bl);
+  if (struct_v >= 2) {
+    decode(l.min_num_shards, bl);
+  } else {
+    l.min_num_shards = 1;
path: src/rgw/rgw_bucket_layout.cc
user: login: cbodley
body: i think we usually assume that we're decoding into a default-constructed object, so don't have to repeat the default values. same with the json decoder
created_at: 2025-02-27T18:44:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -271,16 +277,28 @@ inline uint32_t num_shards(const bucket_index_normal_layout& index) {
   // old buckets used num_shards=0 to mean 1
   return index.num_shards > 0 ? index.num_shards : 1;
 }
+inline uint32_t min_layout_shards(const bucket_index_normal_layout& index) {
+  // old buckets used num_shards=0 to mean 1
+  return index.min_num_shards > 0 ? index.min_num_shards : 1;
path: src/rgw/rgw_bucket_layout.h
user: login: cbodley
body: we won't have to deal with old buckets that set min_num_shards=0, so this can probably just return `index.min_num_shards` directly
created_at: 2025-02-27T18:50:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -8803,6 +8805,51 @@ int main(int argc, const char **argv)
     }
   } // OPT_RESHARD_CANCEL
 
+  if (opt_cmd == OPT::BUCKET_SET_MIN_SHARDS) {
+    if (bucket_name.empty()) {
+      cerr << "ERROR: bucket not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (!num_shards_specified) {
+      cerr << "ERROR: --num-shards not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (num_shards < 1) {
+      cerr << "ERROR: --num-shards must be at least 1" << std::endl;
+      return -EINVAL;
+    }
+
+    int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
+    if (ret < 0) {
+      return -ret;
+    }
+    auto& bucket_info = bucket->get_info();
+
+    const rgw::BucketIndexType type =
+      bucket_info.layout.current_index.layout.type;
+    if (type != rgw::BucketIndexType::Normal) {
+      cerr << "ERROR: the bucket's layout is type " << type <<
+	" instead of type " << rgw::BucketIndexType::Normal <<
+	" and therefore does not have a "
+	"minimum number of shards that can be altered" << std::endl;
+      return -EINVAL;
path: src/rgw/radosgw-admin/radosgw-admin.cc
user: login: cbodley
body: main() returns positive error codes, so `return EINVAL` here
created_at: 2025-02-28T18:49:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -8803,6 +8805,51 @@ int main(int argc, const char **argv)
     }
   } // OPT_RESHARD_CANCEL
 
+  if (opt_cmd == OPT::BUCKET_SET_MIN_SHARDS) {
+    if (bucket_name.empty()) {
+      cerr << "ERROR: bucket not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (!num_shards_specified) {
+      cerr << "ERROR: --num-shards not specified" << std::endl;
+      return -EINVAL;
+    }
+
+    if (num_shards < 1) {
+      cerr << "ERROR: --num-shards must be at least 1" << std::endl;
+      return -EINVAL;
+    }
+
+    int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
+    if (ret < 0) {
+      return -ret;
+    }
+    auto& bucket_info = bucket->get_info();
+
+    const rgw::BucketIndexType type =
+      bucket_info.layout.current_index.layout.type;
+    if (type != rgw::BucketIndexType::Normal) {
+      cerr << "ERROR: the bucket's layout is type " << type <<
+	" instead of type " << rgw::BucketIndexType::Normal <<
+	" and therefore does not have a "
+	"minimum number of shards that can be altered" << std::endl;
+      return -EINVAL;
path: src/rgw/radosgw-admin/radosgw-admin.cc
user: login: ivancich
body: ugh, thanks!
created_at: 2025-02-28T19:35:05Z
author_association: MEMBER
diff_hunk: @@ -6,20 +6,20 @@ RGW Dynamic Bucket Index Resharding
 
 .. versionadded:: Luminous
 
-A large bucket index can lead to performance problems, which can
-be addressed by sharding bucket indexes.
-Until Luminous, changing the number of bucket shards (resharding)
-needed to be done offline, with RGW services disabled.
-Since the Luminous release Ceph has supported online bucket resharding.
+A bucket index object with too many entries can lead to performance
+problems. This can be addressed by resharding bucket indexes.  Until
+Luminous, changing the number of bucket shards (resharding) needed to
path: doc/radosgw/dynamicresharding.rst
user: login: anthonyeleven
body: s/needed to/could only/
created_at: 2025-02-28T19:58:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,6 +54,8 @@ Configuration
 .. confval:: rgw_dynamic_resharding
 .. confval:: rgw_max_objs_per_shard
 .. confval:: rgw_max_dynamic_shards
+.. confval:: rgw_dynamic_resharding_may_reduce
+.. confval:: rgw_dynamic_resharding_reduction_wait
path: doc/radosgw/dynamicresharding.rst
user: login: anthonyeleven
body: Shouldn't this PR add these to `src/common/options/rgw.yaml.in` or is that done elsewhere?
created_at: 2025-02-28T20:01:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,6 +54,8 @@ Configuration
 .. confval:: rgw_dynamic_resharding
 .. confval:: rgw_max_objs_per_shard
 .. confval:: rgw_max_dynamic_shards
+.. confval:: rgw_dynamic_resharding_may_reduce
+.. confval:: rgw_dynamic_resharding_reduction_wait
path: doc/radosgw/dynamicresharding.rst
user: login: cbodley
body: added previously in https://github.com/ceph/ceph/pull/57538
created_at: 2025-02-28T20:19:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,6 +54,8 @@ Configuration
 .. confval:: rgw_dynamic_resharding
 .. confval:: rgw_max_objs_per_shard
 .. confval:: rgw_max_dynamic_shards
+.. confval:: rgw_dynamic_resharding_may_reduce
+.. confval:: rgw_dynamic_resharding_reduction_wait
path: doc/radosgw/dynamicresharding.rst
user: login: ivancich
body: 🙏🏼
created_at: 2025-03-01T00:29:42Z
author_association: MEMBER
diff_hunk: @@ -6,20 +6,20 @@ RGW Dynamic Bucket Index Resharding
 
 .. versionadded:: Luminous
 
-A large bucket index can lead to performance problems, which can
-be addressed by sharding bucket indexes.
-Until Luminous, changing the number of bucket shards (resharding)
-needed to be done offline, with RGW services disabled.
-Since the Luminous release Ceph has supported online bucket resharding.
+A bucket index object with too many entries can lead to performance
+problems. This can be addressed by resharding bucket indexes.  Until
+Luminous, changing the number of bucket shards (resharding) needed to
path: doc/radosgw/dynamicresharding.rst
user: login: ivancich
body: Thank you, @anthonyeleven!
created_at: 2025-03-01T00:33:37Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-01-17T22:40:14Z
body: This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved
author_association: NONE
user: login: ivancich
created_at: 2025-01-21T20:07:21Z
body: Note for anyone reviewing this PR:

Only the final commit has the additional code. The previous commits are from Casey's PR (https://github.com/ceph/ceph/pull/61269), as this PR builds off of it. Once the other PR merges, those other commits can be removed from this PR.

Thanks!
author_association: MEMBER
user: login: ivancich
created_at: 2025-02-04T03:05:34Z
body: > looks good. an admin command like `radosgw-admin bucket set-min-shards` could be useful to support changes to this value

added
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-02-05T18:19:49Z
body: This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved
author_association: NONE
user: login: cbodley
created_at: 2025-02-28T19:38:59Z
body: docs look great 👍 
author_association: CONTRIBUTOR
user: login: ivancich
created_at: 2025-03-01T00:34:24Z
body: QA run:
* https://pulpito.ceph.com/ivancich-2025-02-28_19:54:30-rgw-wip-dynamic-resharding-min-shards-distro-default-smithi/
* https://pulpito.ceph.com/ivancich-2025-02-28_19:53:09-rgw-wip-dynamic-resharding-min-shards-distro-default-smithi/
author_association: MEMBER
user: login: ivancich
created_at: 2025-03-01T05:49:00Z
body: jenkins test make check arm64
author_association: MEMBER