Change:
Add support for the recently emerged feature-rich client to improve support for Ceph RBD in Kubernetes. Kubernetes already offers support for Ceph RBD based on the kernel client. Unfortunately, due to obvious reasons, can't use the user-space space library that gets most of the development focus. This caused a feature gap when compared to VM-based environments. At the moment lacks exclusive locking infrastructure, image management facilities and multi-DC support (remote backups!). The detailed list can be obtained by comparing the definitions of macro in . While there are efforts on partially closing the gap (patches waiting for merge), continuously gets new items like or . Beside the feature gap exhibits additional drawback. Being entirely kernel space impacts fault-tolerance as any kernel panic affects a whole node -- not only a single Pod using RBD storage. According to the the reliability, especially on non-x86 platforms, was one of the driving factors behind the new client. Those issues can be addressed by employing -- a thin adapter () between NBD subsystem of Linux kernel and . The primary concern about was performance. The authors claimed it's comparable to krbd. Together with Intel we verified those claims. Tests on typical HW where Ceph is deployed (SSDs for journals, HDDs as main storage) showed delivers 70-80% of performance. Though, we don't consider the raw performance as a deciding factor. Even fastest RBD client wouldn't be a solution-of-choice for performance critical use cases like RDBMS hosting. We would like to propose incorporating support for . The integration can be accomplished in multiple ways. the current in-tree plugin for RBD to use or depending on a configuration parameter. This would assure the excellent commonality. On the other hand we see differences where this might become an issue. The krbd-based plugin employs the advisory locking mechanism to provide fencing at the cost of HA -- in case of node crash a manual intervention is required. However, serving images marked to use exclusive locking feature would result in employing the same underlying infrastructure twice. At the moment this isn’t an issue because simply doesn’t support exclusive lock. However, when it finally gets it, RBD images created with Ceph’s default capabilities set will be inaccessible in read-write mode. This was the reason why a has been in QEMU. Another problem we see are additional dependencies. After fencing capability is integrated directly in , the dependency on user-space CLI utility can be removed. The helper is not required for mapping RBD images as the kernel interface may be used directly by the plugin. in-tree plugin. Offers greatest flexibility in adopting new features (like dynamic provisioning of Persistent Volumes). FlexVolume plugin. Simple but limited. Bringing support for e.g. dynamic provisioning would require extensive changes in FlexVolume’s interfaces. At the moment the option # 2 looks most promising. Though we stay entirely open for the discussion. efficiency suffers from extra context switches and extensive memory copying between kernel and user-space boundaries. Although we can do nothing with the first factor, the second can be improved for large blocks transfers. Linux kernel has facilities for zero-copy page-aligned memory transfers between kernel and application via the combination of and system calls. Actually Mirantis on bringing fencing to on the top of exclusive lock infrastructure. This would allow to move from the flawed advisory lock mechanism that is currently used by rbd storage plugin. Currently RBD images must be created ahead by a cluster administrator. We see possibility to implement the dynamic provisioning for RBD in the future.
cc especially
is new to the Ceph 2. Existing Ceph 1.x installation doesn't have it. kbrbd is the common denominator. There are other bridging technologies to get librbd to present rbd image as a block device, e.g. tcmu The drawbacks mentioned are being addressed, e.g. to resolve fencing issue.
What do you mean by Ceph 1.x installation? rbd-nbd is part of Jewel release which is supported i.e by ceph-ansible.
I meant the client bits - this is where rbd-nbd is called to map the image.
is still based on the same advisory locking mechanism that has been because of the conflict with flag of an RBD image (enabled by default in Ceph Jewel LTS release). Please take a look on . Ceph community already a dedicated API for fencing. In addition to that I would say that fencing isn't the sole drawback. The feature gap is still there. For instance, what about remote backups and multi-DC support in general?
Agreed, a couple options exist. I recently talked with and in the long term perspective there is a plan to develop TCMU-based bridge for librbd. Anyway, I would expect that those bridges (, , ) will be so similar (as basing on the same library and having the same features including fencing) that it makes sense to address them with one single plugin called e.g. . The abstraction layer would be extremely thin covering basically map and unmap only.
I'm afraid this isn't entirely accurate as the kernel version is a significant degree of freedom. Encountering the version-related limitation tends to be game-over as upgrading the whole kernel to get newer krbd often isn't an option. Moreover, the problem might become much more visible soon as Ilya finishes merging some new features in krbd. User-space solution wouldn't be locked to kernel version to such degree. Changing librbd and an adapter like seems to be much more straightforward and isolated than global kernel upgrade.
Not every bridge is the same. If librbd is implemented as a handler, we need to do the setup. The semantics would be different from rbd-nbd. , Radoslaw Zarzynski
this is why I mentioned the need for the abstraction layer for mapping and unmapping an RBD image.
Ping! Is there any new developments on this?
ping
Ping! Is there any progress here?
I also want this feature
Ping! Any progress?
I've started working on this issue with another team member at Salesforce. We'll update this issue when we have a new PR leveraging rbd-nbd. /assign
As an FYI, I believe there is also an effort just getting underway to implement Container Storage Interface drivers for RBD and CephFS for use by k8s [1] to eventually replace the in-tree drivers. [1]
fyi
Code Freeze begins in 6 hours. Will this feature be completed by then? It needs to be status/approved-for-milestone and at least priority/important-soon to stay open after code freeze. Otherwise, please punt it to 1.11.
do we have any example to test rbd-nbd with k8s?
here is a that I wrote last year after the code was merged on .
I actually started from that blog, but the issue is related to ceph version mismatch I have ceph cluster setup having mimic(13.2) version and the client(rbd-nbd) version is jewel(10.2). After upgrading to rbd-nbd to mimic version the issue is fixed. But as a side not plugin should manage cli change e.g with rbd-nbd (ceph-version 10.2) version check command is and in rbd-nbd (ceph-version 13.2) further details are in this
FYI -- Jewel is EOL otherwise I would fix the parameter parsing since it is incorrect to require to request the version. However, I don't thin k8s should need to work around EOLed software, but perhaps a quick documentation blurb somewhere stating you need Luminous or later might be helpful.

Discussion:
nbdToolsFound := false if !mapped { <del> nbdToolsFound := checkRbdNbdTools(b.exec) </del> <ins> nbdToolsFound = checkRbdNbdTools(b.exec) </ins> if nbdToolsFound { devicePath, mapped = waitForPath(b.Pool, b.Image, 1 /*maxRetries*/, true /*useNbdDriver*/) }