Repository: scylladb/seastar
Pull request #2738: Share IO queues between mountpoints
Author: login: xemul
Created: 2025-04-11T09:02:42Z | Merged: 2025-04-12T20:20:25Z

Currently, IO-queues assignment is done per-mountpoint. The mountpoint paths are read from io-properties.yaml file, for each the underlying device number is stat()-ed, then devive -> io_queue mapping is established. When opening a file, seastar stat()-s the filepath and finds the io_queue for the found device number.

It's nonetheless possible to have one physical disk be "split" into several virtual block devices, and to mount each into its own mountpoint. In order to configure io-scheduler for this setup the only option is to have two mountpoints in io_properties.yaml. With that, seastar will create two io_queues for both mountpoints, but it's not correct, since underlying physical disk is still the same.

This PR extends the io-properties.yaml configuration so that one disk entry in it can specify a list of mountpoints this disk configuration applies to. When met such a record, seastar would create a single io_queue object, but will map all mountpoints' device numbers to it.

The change has two main parts. First, the smaller one, reactor's map<device, io_queue> now may have several devices point to one shared io_queue object. So than when two files are opened on different mountpoints with, respectively, different device numbers, they both will get reference to the same io_queue. Second, the larger change, is that io_queue internal "identification" changes from the underlying device number to abstract integer value. That's not a breaking change, however, device numbers were not present in any APIs or metric labels. Only in logs.

fixes #2733

Linked issue #2733: Teach io-scheduler to merge mountpoints into one io-queue
Sometimes a node is configured to have several virtual volumes (of whatever kind) sitting on the same underlying physical device.
As a result, seastar learns that it needs to use several distinct io-queue-s with its own properties, while in reality the underlying disk is still the same and seastar should have used one shared io-queue for them all

Diff:
diff --git a/apps/io_tester/ioinfo.cc b/apps/io_tester/ioinfo.cc
index ec4e840a91d..613fa7ad1cc 100644
--- a/apps/io_tester/ioinfo.cc
+++ b/apps/io_tester/ioinfo.cc
@@ -58,9 +58,11 @@ int main(int ac, char** av) {
                             auto& ioq = engine().get_io_queue(st.st_dev);
                             auto& cfg = ioq.get_config();
 
+                            out << YAML::Key << "device" << YAML::Value << st.st_dev;
                             out << YAML::Key << "io_latency_goal_ms" << YAML::Value <<
                                     std::chrono::duration_cast<std::chrono::duration<double, std::milli>>(cfg.rate_limit_duration).count();
                             out << YAML::Key << "io_queue" << YAML::BeginMap;
+                            out << YAML::Key << "id" << YAML::Value << ioq.id();
                             out << YAML::Key << "req_count_rate" << YAML::Value << cfg.req_count_rate;
                             out << YAML::Key << "blocks_count_rate" << YAML::Value << cfg.blocks_count_rate;
                             out << YAML::Key << "disk_req_write_to_read_multiplier" << YAML::Value << cfg.disk_req_write_to_read_multiplier;
diff --git a/doc/io-properties-file.md b/doc/io-properties-file.md
index 7503190775d..298ef687d13 100644
--- a/doc/io-properties-file.md
+++ b/doc/io-properties-file.md
@@ -39,3 +39,23 @@ disks:
     write_bandwidth: 510M
     write_saturation_length: 64k
 ```
+
+Optionally, instead of the "mountpoint" there can be the "mountpoints" (plural)
+entry in the list element being a list itself and listing more than one path. As
+a result all the listed mountpoints will share the corresponding internal IO queue.
+
+Example:
+
+```
+disks:
+  - mountpoints:
+      - /var/lib/some_seastar_app/sub_one
+      - /var/lib/some_seastar_app/sub_two
+    read_iops: 95000
+    ...
+```
+
+An example when this configuration is applicable can be an LLVM set of volumes
+from one disk each being mounted at its own path. In that case, different mount
+points would have different (virtual) block devices, yet, they will share the
+same physical disk and thus need to run over one shared IO queue.
diff --git a/include/seastar/core/io_intent.hh b/include/seastar/core/io_intent.hh
index c8d79d636ce..564178fb958 100644
--- a/include/seastar/core/io_intent.hh
+++ b/include/seastar/core/io_intent.hh
@@ -43,12 +43,12 @@ namespace seastar {
 SEASTAR_MODULE_EXPORT
 class io_intent {
     struct intents_for_queue {
-        dev_t dev;
-        io_priority_class_id qid;
+        unsigned qid;
+        io_priority_class_id cid;
         internal::cancellable_queue cq;
 
-        intents_for_queue(dev_t dev_, io_priority_class_id qid_) noexcept
-            : dev(dev_), qid(qid_), cq() {}
+        intents_for_queue(unsigned qid_, io_priority_class_id cid_) noexcept
+            : qid(qid_), cid(cid_), cq() {}
 
         intents_for_queue(intents_for_queue&&) noexcept = default;
         intents_for_queue& operator=(intents_for_queue&&) noexcept = default;
@@ -96,14 +96,14 @@ public:
     }
 
     /// @private
-    internal::cancellable_queue& find_or_create_cancellable_queue(dev_t dev, io_priority_class_id qid) {
+    internal::cancellable_queue& find_or_create_cancellable_queue(unsigned qid, io_priority_class_id cid) {
         for (auto&& i : _intents) {
-            if (i.dev == dev && i.qid == qid) {
+            if (i.qid == qid && i.cid == cid) {
                 return i.cq;
             }
         }
 
-        _intents.emplace_back(dev, qid);
+        _intents.emplace_back(qid, cid);
         return _intents.back().cq;
     }
 };
diff --git a/include/seastar/core/io_queue.hh b/include/seastar/core/io_queue.hh
index 963cdd1cf59..08179b9ee61 100644
--- a/include/seastar/core/io_queue.hh
+++ b/include/seastar/core/io_queue.hh
@@ -81,6 +81,7 @@ public:
 private:
     std::vector<std::unique_ptr<priority_class_data>> _priority_classes;
     io_group_ptr _group;
+    const unsigned _id;
     boost::container::static_vector<fair_queue, 2> _streams;
     internal::io_sink& _sink;
 
@@ -130,7 +131,7 @@ public:
     static constexpr unsigned block_size_shift = 9;
 
     struct config {
-        dev_t devid;
+        unsigned id;
         unsigned long req_count_rate = std::numeric_limits<int>::max();
         unsigned long blocks_count_rate = std::numeric_limits<int>::max();
         unsigned disk_req_write_to_read_multiplier = read_request_base_count;
@@ -180,7 +181,7 @@ public:
     fair_queue_entry::capacity_t request_capacity(internal::io_direction_and_length dnl) const noexcept;
 
     sstring mountpoint() const;
-    dev_t dev_id() const noexcept;
+    unsigned id() const noexcept { return _id; }
 
     void update_shares_for_class(internal::priority_class pc, size_t new_shares);
     future<> update_bandwidth_for_class(internal::priority_class pc, uint64_t new_bandwidth);
@@ -233,10 +234,6 @@ inline sstring io_queue::mountpoint() const {
     return get_config().mountpoint;
 }
 
-inline dev_t io_queue::dev_id() const noexcept {
-    return get_config().devid;
-}
-
 namespace internal {
 double request_tokens(io_direction_and_length dnl, const io_queue::config& cfg) noexcept;
 }
diff --git a/include/seastar/core/reactor.hh b/include/seastar/core/reactor.hh
index de092b4781b..5ac72f16852 100644
--- a/include/seastar/core/reactor.hh
+++ b/include/seastar/core/reactor.hh
@@ -230,7 +230,7 @@ private:
     static constexpr unsigned max_aio = max_aio_per_queue * max_queues;
 
     // Each mountpouint is controlled by its own io_queue, but ...
-    std::unordered_map<dev_t, std::unique_ptr<io_queue>> _io_queues;
+    std::unordered_map<dev_t, seastar::shared_ptr<io_queue>> _io_queues;
     // ... when dispatched all requests get into this single sink
     internal::io_sink _io_sink;
     unsigned _num_io_groups = 0;
diff --git a/include/seastar/core/resource.hh b/include/seastar/core/resource.hh
index 58dac54b4ff..d97f4c92d05 100644
--- a/include/seastar/core/resource.hh
+++ b/include/seastar/core/resource.hh
@@ -23,6 +23,7 @@
 
 #include <seastar/util/spinlock.hh>
 #include <seastar/util/modules.hh>
+#include <seastar/core/shared_ptr.hh>
 #ifndef SEASTAR_MODULE
 #include <cassert>
 #include <cstdlib>
@@ -96,7 +97,7 @@ struct configuration {
     size_t cpus;
     cpuset cpu_set;
     bool assign_orphan_cpus = false;
-    std::vector<dev_t> devices;
+    std::vector<unsigned> io_queues;
     unsigned num_io_groups;
     hwloc::internal::topology_holder topology;
 };
@@ -108,7 +109,7 @@ struct memory {
 };
 
 struct io_queue_topology {
-    std::vector<std::unique_ptr<io_queue>> queues;
+    std::vector<seastar::shared_ptr<io_queue>> queues;
     std::vector<unsigned> shard_to_group;
     std::vector<unsigned> shards_in_group;
     std::vector<std::shared_ptr<io_group>> groups;
@@ -128,7 +129,7 @@ struct cpu {
 
 struct resources {
     std::vector<cpu> cpus;
-    std::unordered_map<dev_t, io_queue_topology> ioq_topology;
+    std::unordered_map<unsigned, io_queue_topology> ioq_topology;
     std::unordered_map<unsigned /* numa node id */, cpuset> numa_node_id_to_cpuset;
 };
 
diff --git a/src/core/file-impl.hh b/src/core/file-impl.hh
index 89eec92f20a..6d32f8e1fa6 100644
--- a/src/core/file-impl.hh
+++ b/src/core/file-impl.hh
@@ -73,6 +73,7 @@ public:
 class posix_file_impl : public file_impl {
     std::atomic<unsigned>* _refcount = nullptr;
     const bool _nowait_works;
+    const dev_t _device_id;
     io_queue& _io_queue;
     const open_flags _open_flags;
 protected:
diff --git a/src/core/file.cc b/src/core/file.cc
index 468d2df7fe3..3fac7033c20 100644
--- a/src/core/file.cc
+++ b/src/core/file.cc
@@ -116,7 +116,8 @@ file_handle::to_file() && {
 
 posix_file_impl::posix_file_impl(int fd, open_flags f, file_open_options options, dev_t device_id, bool nowait_works)
         : _nowait_works(nowait_works)
-        , _io_queue(engine().get_io_queue(device_id))
+        , _device_id(device_id)
+        , _io_queue(engine().get_io_queue(_device_id))
         , _open_flags(f)
         , _fd(fd)
 {
@@ -165,7 +166,7 @@ posix_file_impl::dup() {
     if (!_refcount) {
         _refcount = new std::atomic<unsigned>(1u);
     }
-    auto ret = std::make_unique<posix_file_handle_impl>(_fd, _open_flags, _refcount, _io_queue.dev_id(),
+    auto ret = std::make_unique<posix_file_handle_impl>(_fd, _open_flags, _refcount, _device_id,
             _memory_dma_alignment, _disk_read_dma_alignment, _disk_write_dma_alignment, _disk_overwrite_dma_alignment,
             _nowait_works);
     _refcount->fetch_add(1, std::memory_order_relaxed);
@@ -180,7 +181,8 @@ posix_file_impl::posix_file_impl(int fd, open_flags f, std::atomic<unsigned>* re
         bool nowait_works)
         : _refcount(refcount)
         , _nowait_works(nowait_works)
-        , _io_queue(engine().get_io_queue(device_id))
+        , _device_id(device_id)
+        , _io_queue(engine().get_io_queue(_device_id))
         , _open_flags(f)
         , _fd(fd) {
     _memory_dma_alignment = memory_dma_alignment;
diff --git a/src/core/io_queue.cc b/src/core/io_queue.cc
index 25219baafc6..5f5ac7cc556 100644
--- a/src/core/io_queue.cc
+++ b/src/core/io_queue.cc
@@ -230,11 +230,11 @@ class io_desc_read_write final : public io_completion {
         , _fq_capacity(cap)
         , _iovs(std::move(iovs))
     {
-        io_log.trace("dev {} : req {} queue  len {} capacity {}", _ioq.dev_id(), fmt::ptr(this), _dnl.length(), _fq_capacity);
+        io_log.trace("dev {} : req {} queue  len {} capacity {}", _ioq.id(), fmt::ptr(this), _dnl.length(), _fq_capacity);
     }
 
     virtual void set_exception(std::exception_ptr eptr) noexcept override {
-        io_log.trace("dev {} : req {} error", _ioq.dev_id(), fmt::ptr(this));
+        io_log.trace("dev {} : req {} error", _ioq.id(), fmt::ptr(this));
         _pclass.on_error();
         _ioq.complete_request(*this, std::chrono::duration<double>(0.0));
         _pr.set_exception(eptr);
@@ -242,7 +242,7 @@ class io_desc_read_write final : public io_completion {
     }
 
     virtual void complete(size_t res) noexcept override {
-        io_log.trace("dev {} : req {} complete", _ioq.dev_id(), fmt::ptr(this));
+        io_log.trace("dev {} : req {} complete", _ioq.id(), fmt::ptr(this));
         auto now = io_queue::clock_type::now();
         auto delay = std::chrono::duration_cast<std::chrono::duration<double>>(now - _ts);
         _pclass.on_complete(delay);
@@ -258,7 +258,7 @@ class io_desc_read_write final : public io_completion {
     }
 
     void dispatch() noexcept {
-        io_log.trace("dev {} : req {} submit", _ioq.dev_id(), fmt::ptr(this));
+        io_log.trace("dev {} : req {} submit", _ioq.id(), fmt::ptr(this));
         auto now = io_queue::clock_type::now();
         _pclass.on_dispatch(_dnl, std::chrono::duration_cast<std::chrono::duration<double>>(now - _ts));
         _ts = now;
@@ -568,6 +568,7 @@ fair_queue::config io_queue::make_fair_queue_config(const config& iocfg, sstring
 io_queue::io_queue(io_group_ptr group, internal::io_sink& sink)
     : _priority_classes()
     , _group(std::move(group))
+    , _id(_group->_config.id)
     , _sink(sink)
     , _averaging_decay_timer([this] {
         update_flow_ratio();
@@ -600,7 +601,7 @@ io_queue::io_queue(io_group_ptr group, internal::io_sink& sink)
 
 fair_group::config io_group::make_fair_group_config(const io_queue::config& qcfg) noexcept {
     fair_group::config cfg;
-    cfg.label = fmt::format("io-queue-{}", qcfg.devid);
+    cfg.label = fmt::format("io-queue-{}", qcfg.id);
     double min_weight = std::min(io_queue::read_request_base_count, qcfg.disk_req_write_to_read_multiplier);
     double min_size = std::min(io_queue::read_request_base_count, qcfg.disk_blocks_write_to_read_multiplier);
     cfg.min_tokens = min_weight / qcfg.req_count_rate + min_size / qcfg.blocks_count_rate;
@@ -627,7 +628,7 @@ io_group::io_group(io_queue::config io_cfg, unsigned nr_queues)
 
     auto goal = io_latency_goal();
     auto lvl = goal > 1.1 * _config.rate_limit_duration ? log_level::warn : log_level::debug;
-    seastar_logger.log(lvl, "IO queue uses {:.2f}ms latency goal for device {}", goal.count() * 1000, _config.devid);
+    seastar_logger.log(lvl, "IO queue uses {:.2f}ms latency goal for {}", goal.count() * 1000, _config.mountpoint);
 
     /*
      * The maximum request size shouldn't result in the capacity that would
@@ -864,7 +865,7 @@ future<size_t> io_queue::queue_one_request(internal::priority_class pc, io_direc
         auto queued_req = std::make_unique<queued_io_request>(std::move(req), *this, cap, pclass, std::move(dnl), std::move(iovs));
         auto fut = queued_req->get_future();
         if (intent != nullptr) {
-            auto& cq = intent->find_or_create_cancellable_queue(dev_id(), pc.id());
+            auto& cq = intent->find_or_create_cancellable_queue(_id, pc.id());
             queued_req->set_intent(cq);
         }
 
diff --git a/src/core/reactor.cc b/src/core/reactor.cc
index 7b52742fbfa..c4e1b3e8e41 100644
--- a/src/core/reactor.cc
+++ b/src/core/reactor.cc
@@ -33,6 +33,7 @@ module;
 #include <ranges>
 #include <regex>
 #include <thread>
+#include <unordered_set>
 
 #include <grp.h>
 #include <spawn.h>
@@ -189,8 +190,9 @@ static_assert(posix::shutdown_mask(SHUT_RD) == posix::rcv_shutdown);
 static_assert(posix::shutdown_mask(SHUT_WR) == posix::snd_shutdown);
 static_assert(posix::shutdown_mask(SHUT_RDWR) == (posix::snd_shutdown | posix::rcv_shutdown));
 
-struct mountpoint_params {
-    std::string mountpoint = "none";
+struct disk_params {
+    std::vector<std::string> mountpoints;
+    std::vector<dev_t> devices;
     uint64_t read_bytes_rate = std::numeric_limits<uint64_t>::max();
     uint64_t write_bytes_rate = std::numeric_limits<uint64_t>::max();
     uint64_t read_req_rate = std::numeric_limits<uint64_t>::max();
@@ -205,10 +207,14 @@ struct mountpoint_params {
 
 namespace YAML {
 template<>
-struct convert<seastar::mountpoint_params> {
-    static bool decode(const Node& node, seastar::mountpoint_params& mp) {
+struct convert<seastar::disk_params> {
+    static bool decode(const Node& node, seastar::disk_params& mp) {
         using namespace seastar;
-        mp.mountpoint = node["mountpoint"].as<std::string>().c_str();
+        if (node["mountpoints"]) {
+            mp.mountpoints = node["mountpoints"].as<std::vector<std::string>>();
+        } else {
+            mp.mountpoints.push_back(node["mountpoint"].as<std::string>());
+        }
         mp.read_bytes_rate = parse_memory_size(node["read_bandwidth"].as<std::string>());
         mp.read_req_rate = parse_memory_size(node["read_iops"].as<std::string>());
         mp.write_bytes_rate = parse_memory_size(node["write_bandwidth"].as<std::string>());
@@ -4080,7 +4086,7 @@ class disk_config_params {
 private:
     const unsigned _max_queues;
     unsigned _num_io_groups = 0;
-    std::unordered_map<dev_t, mountpoint_params> _mountpoints;
+    std::unordered_map<unsigned, disk_params> _disks;
     std::chrono::duration<double> _latency_goal;
     std::chrono::milliseconds _stall_threshold;
     double _flow_ratio_backpressure_threshold;
@@ -4144,21 +4150,28 @@ class disk_config_params {
                 if (sec_name != "disks") {
                     throw std::runtime_error(fmt::format("While parsing I/O options: section {} currently unsupported.", sec_name));
                 }
-                auto disks = section.second.as<std::vector<mountpoint_params>>();
+                auto disks = section.second.as<std::vector<disk_params>>();
+                unsigned queue_id_gen = 1;
+                std::unordered_set<dev_t> devices;
                 for (auto& d : disks) {
-                    struct ::stat buf;
-                    auto ret = stat(d.mountpoint.c_str(), &buf);
-                    if (ret < 0) {
-                        throw std::runtime_error(fmt::format("Couldn't stat {}", d.mountpoint));
+                    for (auto mp : d.mountpoints) {
+                        struct ::stat buf;
+                        auto ret = stat(mp.c_str(), &buf);
+                        if (ret < 0) {
+                            throw std::runtime_error(fmt::format("Couldn't stat {}", mp));
+                        }
+
+                        auto st_dev = S_ISBLK(buf.st_mode) ? buf.st_rdev : buf.st_dev;
+                        d.devices.push_back(st_dev);
+                        auto [ it, inserted ] = devices.insert(st_dev);
+                        if (!inserted) {
+                            throw std::runtime_error(fmt::format("Mountpoint {}, device {} already configured", mp, st_dev));
+                        }
                     }
 
-                    auto st_dev = S_ISBLK(buf.st_mode) ? buf.st_rdev : buf.st_dev;
-                    if (_mountpoints.count(st_dev)) {
-                        throw std::runtime_error(fmt::format("Mountpoint {} already configured", d.mountpoint));
-                    }
-                    if (_mountpoints.size() >= _max_queues) {
+                    if (_disks.size() >= _max_queues) {
                         throw std::runtime_error(fmt::format("Configured number of queues {} is larger than the maximum {}",
-                                                 _mountpoints.size(), _max_queues));
+                                                 _disks.size(), _max_queues));
                     }
 
                     d.read_bytes_rate *= d.rate_factor;
@@ -4171,23 +4184,25 @@ class disk_config_params {
                         throw std::runtime_error(fmt::format("R/W bytes and req rates must not be zero"));
                     }
 
-                    seastar_logger.debug("dev_id: {} mountpoint: {}", st_dev, d.mountpoint);
-                    _mountpoints.emplace(st_dev, d);
+                    unsigned q = queue_id_gen++;
+                    seastar_logger.debug("queue-id: {} mountpoints: {} devices: {}", q, d.mountpoints, d.devices);
+                    _disks.emplace(q, d);
                 }
             }
         }
 
         // Placeholder for unconfigured disks.
-        mountpoint_params d = {};
-        _mountpoints.emplace(0, d);
+        disk_params d = {};
+        d.devices.push_back(0);
+        _disks.emplace(0, d);
     }
 
-    struct io_queue::config generate_config(dev_t devid, unsigned nr_groups) const {
-        seastar_logger.debug("generate_config dev_id: {}", devid);
-        const mountpoint_params& p = _mountpoints.at(devid);
+    struct io_queue::config generate_config(unsigned q, unsigned nr_groups) const {
+        const disk_params& p = _disks.at(q);
+        seastar_logger.debug("generate_config queue-id: {}", q);
         struct io_queue::config cfg;
 
-        cfg.devid = devid;
+        cfg.id = q;
 
         if (p.read_bytes_rate != std::numeric_limits<uint64_t>::max()) {
             cfg.blocks_count_rate = (io_queue::read_request_base_count * (unsigned long)per_io_group(p.read_bytes_rate, nr_groups)) >> io_queue::block_size_shift;
@@ -4203,7 +4218,7 @@ class disk_config_params {
         if (p.write_saturation_length != std::numeric_limits<uint64_t>::max()) {
             cfg.disk_write_saturation_length = p.write_saturation_length;
         }
-        cfg.mountpoint = p.mountpoint;
+        cfg.mountpoint = fmt::to_string(fmt::join(p.mountpoints, ":"));
         cfg.duplex = p.duplex;
         cfg.rate_limit_duration = latency_goal();
         cfg.flow_ratio_backpressure_threshold = _flow_ratio_backpressure_threshold;
@@ -4217,8 +4232,12 @@ class disk_config_params {
         return cfg;
     }
 
-    auto device_ids() {
-        return boost::adaptors::keys(_mountpoints);
+    auto queue_ids() {
+        return boost::adaptors::keys(_disks);
+    }
+
+    const std::vector<dev_t>& queue_devices(unsigned q) const {
+        return _disks.at(q).devices;
     }
 };
 
@@ -4422,8 +4441,8 @@ void smp::configure(const smp_options& smp_opts, const reactor_options& reactor_
 
     disk_config_params disk_config(reactor::max_queues);
     disk_config.parse_config(smp_opts, reactor_opts);
-    for (auto& id : disk_config.device_ids()) {
-        rc.devices.push_back(id);
+    for (auto& id : disk_config.queue_ids()) {
+        rc.io_queues.push_back(id);
     }
     rc.num_io_groups = disk_config.num_io_groups();
 
@@ -4530,7 +4549,14 @@ void smp::configure(const smp_options& smp_opts, const reactor_options& reactor_
     //     also pre-resize()-d in advance)
 
     auto alloc_io_queues = [&ioq_topology, &disk_config] (shard_id shard) {
-        for (auto& [dev, io_info] : ioq_topology) {
+        for (auto& [q, io_info] : ioq_topology) {
+            auto num_io_groups = io_info.groups.size();
+            if (engine()._num_io_groups == 0) {
+                engine()._num_io_groups = num_io_groups;
+            } else if (engine()._num_io_groups != num_io_groups) {
+                throw std::logic_error(format("Number of IO-groups mismatch, {} != {}", engine()._num_io_groups, num_io_groups));
+            }
+
             auto group_idx = io_info.shard_to_group[shard];
             std::shared_ptr<io_group> group;
 
@@ -4538,29 +4564,24 @@ void smp::configure(const smp_options& smp_opts, const reactor_options& reactor_
                 std::lock_guard _(io_info.lock);
                 auto& iog = io_info.groups[group_idx];
                 if (!iog) {
-                    struct io_queue::config qcfg = disk_config.generate_config(dev, io_info.groups.size());
+                    struct io_queue::config qcfg = disk_config.generate_config(q, io_info.groups.size());
                     iog = std::make_shared<io_group>(std::move(qcfg), io_info.shards_in_group[group_idx]);
-                    seastar_logger.debug("allocate {} IO group with {} queues, dev {}", group_idx, io_info.shards_in_group[group_idx], dev);
+                    seastar_logger.debug("allocate {} IO group with {} queues, queue-id {}", group_idx, io_info.shards_in_group[group_idx], q);
                 }
                 group = iog;
             }
 
-            io_info.queues[shard] = std::make_unique<io_queue>(std::move(group), engine()._io_sink);
-            seastar_logger.debug("attached {} queue to {} IO group, dev {}", shard, group_idx, dev);
+            io_info.queues[shard] = seastar::make_shared<io_queue>(std::move(group), engine()._io_sink);
+            seastar_logger.debug("attached {} queue to {} IO group, queue-id {}", shard, group_idx, q);
         }
     };
 
-    auto assign_io_queues = [&ioq_topology] (shard_id shard) {
-        for (auto& [dev, io_info] : ioq_topology) {
+    auto assign_io_queues = [&ioq_topology, &disk_config] (shard_id shard) {
+        for (auto& [q, io_info] : ioq_topology) {
             auto queue = std::move(io_info.queues[shard]);
             SEASTAR_ASSERT(queue);
-            engine()._io_queues.emplace(dev, std::move(queue));
-
-            auto num_io_groups = io_info.groups.size();
-            if (engine()._num_io_groups == 0) {
-                engine()._num_io_groups = num_io_groups;
-            } else if (engine()._num_io_groups != num_io_groups) {
-                throw std::logic_error(format("Number of IO-groups mismatch, {} != {}", engine()._num_io_groups, num_io_groups));
+            for (const dev_t& dev : disk_config.queue_devices(q)) {
+                engine()._io_queues.emplace(dev, queue);
             }
         }
     };
diff --git a/src/core/resource.cc b/src/core/resource.cc
index 12593cb8104..ffe13b855c4 100644
--- a/src/core/resource.cc
+++ b/src/core/resource.cc
@@ -715,8 +715,8 @@ resources allocate(configuration& c) {
     }
 
     unsigned last_node_idx = 0;
-    for (auto devid : c.devices) {
-        ret.ioq_topology.emplace(devid, allocate_io_queues(topology, ret.cpus, cpu_to_node, c.num_io_groups, last_node_idx));
+    for (auto q : c.io_queues) {
+        ret.ioq_topology.emplace(q, allocate_io_queues(topology, ret.cpus, cpu_to_node, c.num_io_groups, last_node_idx));
     }
 
     ret.numa_node_id_to_cpuset = numa_node_id_to_cpuset(topology);


Reviews:
user: login: mykaul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T07:09:26Z
user: login: tchaikov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T08:23:12Z

Review comments:
diff_hunk: @@ -39,3 +39,23 @@ disks:
     write_bandwidth: 510M
     write_saturation_length: 64k
 ```
+
+Optionally, instead of the "mountpoint" there can be the "mountpoints" (plural)
+entry in the list element being a list itself and listing more than one path. As
+a result all the listed mountpoints will share the corresponding internal IO queue.
+
+Example:
+
+```
+disks:
+  - mountpoints:
+      - /var/lib/some_seastar_app/sub_one
+      - /var/lib/some_seastar_app/sub_two
+    read_iops: 95000
+    ...
+```
+
+An example when this configuration is applicable can be an LLVM set of volumes
path: doc/io-properties-file.md
user: login: mykaul
body: LVM, not LLVM.
created_at: 2025-04-15T07:09:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,3 +39,23 @@ disks:
     write_bandwidth: 510M
     write_saturation_length: 64k
 ```
+
+Optionally, instead of the "mountpoint" there can be the "mountpoints" (plural)
+entry in the list element being a list itself and listing more than one path. As
+a result all the listed mountpoints will share the corresponding internal IO queue.
+
+Example:
+
+```
+disks:
+  - mountpoints:
+      - /var/lib/some_seastar_app/sub_one
+      - /var/lib/some_seastar_app/sub_two
+    read_iops: 95000
+    ...
+```
+
+An example when this configuration is applicable can be an LLVM set of volumes
path: doc/io-properties-file.md
user: login: tchaikov
body: created #2748
created_at: 2025-04-18T08:23:11Z
author_association: CONTRIBUTOR

PR comments:
user: login: ruthea
created_at: 2025-04-11T19:31:32Z
body: Thank you @xemul !
author_association: NONE
user: login: regevran
created_at: 2025-04-20T07:57:47Z
body: @xemul - please [backport](https://github.com/scylladb/scylla-enterprise/issues/5353#issuecomment-2817042269) to 2024.1
author_association: NONE
user: login: bhalevy
created_at: 2025-04-24T11:18:56Z
body: > @xemul - please [backport](https://github.com/scylladb/scylla-enterprise/issues/5353#issuecomment-2817042269) to 2024.1

@xemul ping 
Also, how about 2024.2?
author_association: MEMBER
user: login: regevran
created_at: 2025-04-24T11:28:18Z
body: @avikivity  [said](https://github.com/scylladb/scylla-enterprise/issues/5353#issuecomment-2817102992) he will backport - please coordinate with him.
author_association: NONE