NVMe over Fabrics

NVMe over Fabrics
(NVMe-oF) allows sending NVMe commands over
Ethernet
or
Fibre Channel
. This can be used for remote access to block devices similar to
iSCSI
. An NVMe-oF
host
can access devices exposed by an NVMe-oF
controller
. This is not limited to NVMe devices, you can also expose
block devices
such as
ZFS
zvols.
Host configuration
To access a block device exposed by a remote NVMe-oF controller,
install
the
nvme-cli
package.
First, make sure the necessary kernel module is loaded:
# modprobe nvme-fabrics
Now you can use the CLI to connect to the device:
# nvme connect --transport=tcp --traddr=192.168.0.5 --trsvcid=4420 --nqn=nqn.2024-08.com.example:my-disk
--transport=tcp
Specifies the transport protocol. TCP is the most simple to set up.
--traddr=192.168.0.5
The address of the NVMe-oF target, i.e. the server that exposes the block device.
--trsvcid=4420
The port the target is listening on. 4420 is the recommended port (IANA assignment).
--nqn=nqn.2024-08.com.examplemy-disk
The NVMe Qualified Name (NQN) of the
device
to connect to. The device must be configured on the controller.
After running this command, the device will be available as a normal
NVMe block device
, e.g. under
/dev/nvme0n1
with partitions under
/dev/nvme0n1p1
. It is recommended to refer to these devices by UUID, however.
Controller configuration
Install
the
nvmetcli
package.
First, make sure the necessary kernel module is loaded:
# modprobe nvmet
Controller configuration happens through a file system located at
/sys/kernel/config/nvmet
.
nvmetcli
provides a convenient interface to modify that file system. It also provides a way to load and save the settings.
nvmetcli save
will save the current state to
/etc/nvmet/config.json
, and
nvmetcli restore
will load from that location. You can
enable
the
nvmet.service
unit to load the configuration from that file automatically.
Warning
nvmetcli
will only save, load and edit settings that it recognizes. If you configure unsupported settings (such as DHCHAP) they will be silently dropped. You will have to write them manually to the above filesystem.
Inside
nvmetcli
, you can navigate using
cd
and show the config tree using
ls
. There are three top-level directories.
hosts
contains the NQNs of hosts that are referenced in other parts of the tree, e.g. for access control. You can add an NQN here and then make the client use that NQN using the
nvme connect --hostnqn=...
parameter.
ports
configures the protocols used to access the devices, e.g. a TCP port.
subsystems
configures the individual accessible devices.
Adding a device
First you'll want to create a device to expose:
/> cd subsystems
/subsystems> create nqn.2024-08.com.example:my-device
/subsystems> cd nqn.2024-08.com.example:my-device
/subsystems/n...ple:my-device>
Configure access for the device:
/subsystems/n...ple:my-device> set attr allow_any_host=1
Warning
This will allow any host with network access to this controller to access the block device. There is no further access control by default.
Configure a namespace for the device and set the backing block device:
/subsystems/n...ple:my-device> cd namespaces
/subsystems/n...ce/namespaces> create 1
/subsystems/n...ce/namespaces> cd 1
/subsystems/n.../namespaces/1> set device path=/dev/path/to/block/device
/subsystems/n.../namespaces/1> enable
Configuring the port
Next, create a port. Navigate back to the top using
cd /
.
/> cd ports
/ports> create 1
/ports> cd 1
/ports/1> set addr trtype=tcp traddr=192.168.0.5 trsvcid=4420 adrfam=ipv4
trtype
The transport protocol
traddr
The listener address
trsvcid
The listener TCP port ("service ID")
Add the device you created above to the network port:
/ports/1> cd subsystems
/ports/1/subsystems> create nqn.2024-08.com.example:my-device
That's it! Now the device is accessible to the host using the above instructions. To make this permanent, give the
saveconfig
command and ensure
nvmet.service
is enabled/started.
Authentication using DHCHAP
It is prudent to not allow just any device access to the exposed block device. To authenticate the host, NVMe-oF offers a DHCHAP handshake using a secret shared between host and controller. Unfortunately, as of writing,
nvmetcli
does not support this out of the box, so you have to modify the config FS manually. The host
nvme
command does support DHCHAP.
First, use
nvmetcli
to limit access to the device to only a single host, identified by NQN.
/> cd hosts
/hosts> create nqn.2024-08.com.example.host
/hosts> cd /subsystems/nqn.2024-08.com.example:my-device
/subsystems/n...ple:my-device> set attr allow_any_host=0
/subsystems/n...ple:my-device> cd allowed_hosts
/subsystems/n...allowed_hosts> create nqn.2024-08.com.example.host
The NVMe controller now knows that only the host with the NQN
nqn.2024-08.com.example.host
may access the device. You can test this (without DHCHAP, for now) using
nvme connect --hostnqn=nqn.2024-08.com.example.host
. With the wrong NQN, the connection should fail, with the right NQN, it should succeed.
The next step is to configure DHCHAP. First you need to generate a secret like so:
$ nvme gen-dhchap-key --nqn=nqn.2024-08.com.example.host
DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:
Because
nvmetcli
does not support DHCHAP, you must configure it manually:
# echo 'DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:' > /sys/kernel/config/nvmet/hosts/nqn.2024-08.com.example.host/dhchap_key
Now, you can connect using the DHCHAP secret:
# nvme connect --transport=tcp --traddr=192.168.0.5 --trsvcid=4420 --nqn=nqn.2024-08.com.example:my-disk --hostnqn=nqn.2024-08.com.example.host --dhchap-secret="DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:"
Warning
DHCHAP will authenticate the host but not encrypt the block device data on the network.
Booting from NVMe-oF
It is possible to connect to an NVMe-oF device during the boot process inside
initramfs
and use the device for the root file system. The procedure is very similar to that for
iSCSI
. Only the NVMe-specific parts are listed here for now.
The
initcpio
hook is different for NVMe-oF. The install script adds different modules, and adds the
nvme
binary instead of
iscsistart
:
/etc/initcpio/install/nvme-of
build () {
	map add_module nvme-fabrics nvme-tcp nvme-keyring
	add_checked_modules "/drivers/net"
	add_binary nvme
	add_runscript
}
help () {
	cat <<HELPEOF
	This hook allows you to boot from an NVMe-oF target.
HELPEOF
}
The actual hook uses the CLI to connect to the device:
/etc/initcpio/hooks/nvme-of
run_hook () {
	msg "Mounting NVMe-oF target"
	nvme connect --transport=tcp --nqn=nqn.2024-08.com.example:my-device --traddr=192.168.0.5 --trsvcid=4420
}
Tip
If you configured DHCHAP, make sure to add the required arguments here.
Modify the
mkinitcpio
configuration:
/etc/mkinitcpio.conf
MODULES=(... nvme-fabrics)
HOOKS=(... net nvme-of block ...)
Troubleshooting
Page allocation failure
If you are seeing this failure on the host:
nvme nvme0: Connect command failed, error wo/DNR bit: 6
nvme nvme0: failed to connect queue: 3 ret=6
could not add new controller: failed to write to nvme-fabrics device
Check on the controller:
dmesg
kworker/5:1H: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 5 PID: ... Comm: kworker/5:1H Tainted: P           OE      ...
Hardware name: ...
Workqueue: nvmet_tcp_wq nvmet_tcp_io_work [nvmet_tcp]
Call Trace:
 <TASK>
 dump_stack_lvl+0x4d/0x70
 warn_alloc+0x165/0x1e0
 ? __alloc_pages_direct_compact+0x163/0x390
 __alloc_pages_slowpath.constprop.0+0xce9/0xde0
 __alloc_pages+0x320/0x340
 ? nvmet_tcp_install_queue+0x50/0x140 [nvmet_tcp ...]
 __kmalloc_large_node+0x71/0x130
 __kmalloc+0xc4/0x130
 nvmet_tcp_install_queue+0x50/0x140 [nvmet_tcp ...]
 nvmet_install_queue+0xa6/0x1f0 [nvmet ...]
 nvmet_execute_io_connect+0xd1/0x1a0 [nvmet ...]
 nvmet_tcp_io_work+0x811/0x880 [nvmet_tcp ...]
 process_one_work+0x180/0x350
 worker_thread+0x315/0x450
 ? __pfx_worker_thread+0x10/0x10
 kthread+0xe8/0x120
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x34/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>
If the stack traces match, you are running into a
linux-nvme bug
. To serve a new TCP connection, the NVMe target / controller allocates a relatively large contiguous buffer in kernel space. If kernel memory is full (or fragmented), such a large contiguous region may not be available. This used to cause a kernel crash that was fixed in recent kernels, but as of February 2025 this still prevents new TCP connections to the controller. A patch for that is pending but
stuck in review
.
To work around this issue, you can use the following command to free up kernel memory on the controller prior to connection:
# echo 3 > /proc/sys/vm/drop_caches
Another option that seems to work is to create and delete a large file in tmpfs:
# dd if=/dev/urandom of=/tmp/random-buffer bs=1M count=4096 status=progress; rm -f /tmp/random-buffer
If that does not help, a reboot might be necessary.