Chapter 15. Configuring NVMe over fabrics using NVMe/RDMA In an Non-volatile Memory ExpressTM (NVMeTM) over RDMA (NVMeTM/RDMA) setup, you configure an NVMe controller and an NVMe initiator. 15.1. Setting up an NVMe/RDMA controller using configfs You can configure an Non-volatile Memory ExpressTM (NVMeTM) over RDMA (NVMeTM/RDMA) controller by using configfs . Prerequisites Verify that you have a block device to assign to the nvmet subsystem. Procedure Create the nvmet-rdma subsystem: Replace testnqn with the subsystem name. Allow any host to connect to this controller: Configure a namespace: Replace 10 with the namespace number Set a path to the NVMe device: Enable the namespace: Create a directory with an NVMe port: Display the IP address of mlx5_ib0 : Set the transport address for the controller: Set RDMA as the transport type: Set the address family for the port: Create a soft link: Verification Verify that the NVMe controller is listening on the given port and ready for connection requests: Additional resources nvme(1) man page on your system 15.2. Setting up the NVMe/RDMA controller using nvmetcli You can configure the Non-volatile Memory ExpressTM (NVMeTM) over RDMA (NVMeTM/RDMA) controller by using the nvmetcli utility. The nvmetcli utility provides a command line and an interactive shell option. Prerequisites Verify that you have a block device to assign to the nvmet subsystem. Execute the following nvmetcli operations as a root user. Procedure Install the nvmetcli package: Download the rdma.json file: Edit the rdma.json file and change the traddr value to 172.31.0.202 . Setup the controller by loading the NVMe controller configuration file: Note If the NVMe controller configuration file name is not specified, the nvmetcli uses the /etc/nvmet/config.json file. Verification Verify that the NVMe controller is listening on the given port and ready for connection requests: Optional: Clear the current NVMe controller: Additional resources nvmetcli and nvme(1) man pages on your system 15.3. Configuring an NVMe/RDMA host You can configure a Non-volatile Memory ExpressTM (NVMeTM) over RDMA (NVMeTM/RDMA) host by using the NVMe management command-line interface ( nvme-cli ) tool. Procedure Install the nvme-cli tool: Load the nvme-rdma module if it is not loaded: Discover available subsystems on the NVMe controller: Connect to the discovered subsystems: Replace testnqn with the NVMe subsystem name. Replace 172.31.0.202 with the controller IP address. Replace 4420 with the port number. Verification List the NVMe devices that are currently connected: Optional: Disconnect from the controller: Additional resources nvme(1) man page on your system 15.4. steps Enabling multipathing on NVMe devices

Commands:
modprobe nvmet-rdma mkdir /sys/kernel/config/nvmet/subsystems/ testnqn cd /sys/kernel/config/nvmet/subsystems/ testnqn
echo 1 > attr_allow_any_host
mkdir namespaces/ 10 cd namespaces/ 10
echo -n /dev/ nvme0n1 > device_path
echo 1 > enable
mkdir /sys/kernel/config/nvmet/ports/ 1 cd /sys/kernel/config/nvmet/ports/ 1
ip addr show mlx5_ib0 8: mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP group default qlen 256 link/infiniband 00:00:06:2f:fe:80:00:00:00:00:00:00:e4:1d:2d:03:00:e7:0f:f6 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 172.31.0.202/24 brd 172.31.0.255 scope global noprefixroute mlx5_ib0 valid_lft forever preferred_lft forever inet6 fe80::e61d:2d03:e7:ff6/64 scope link noprefixroute valid_lft forever preferred_lft forever
echo -n 172.31.0.202 > addr_traddr
echo rdma > addr_trtype echo 4420 > addr_trsvcid
echo ipv4 > addr_adrfam
ln -s /sys/kernel/config/nvmet/subsystems/ testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ testnqn
dmesg | grep "enabling port" [ 1091.413648] nvmet_rdma: enabling port 1 (172.31.0.202:4420)
yum install nvmetcli
wget http://git.infradead.org/users/hch/nvmetcli.git/blob_plain/0a6b088db2dc2e5de11e6f23f1e890e4b54fee64:/rdma.json
nvmetcli restore rdma.json
dmesg | tail -1 [ 4797.132647] nvmet_rdma: enabling port 2 (172.31.0.202:4420)
nvmetcli clear
yum install nvme-cli
modprobe nvme-rdma
nvme discover -t rdma -a 172.31.0.202 -s 4420 Discovery Log Number of Records 1, Generation counter 2 =====Discovery Log Entry 0====== trtype: rdma adrfam: ipv4 subtype: nvme subsystem treq: not specified, sq flow control disable supported portid: 1 trsvcid: 4420 subnqn: testnqn traddr: 172.31.0.202 rdma_prtype: not specified rdma_qptype: connected rdma_cms: rdma-cm rdma_pkey: 0x0000
nvme connect -t rdma -n testnqn -a 172.31.0.202 -s 4420 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home nvme0n1 cat /sys/class/nvme/nvme0/transport rdma
nvme list
nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─rhel_rdma--virt--03-root 253:0 0 50G 0 lvm / ├─rhel_rdma--virt--03-swap 253:1 0 4G 0 lvm [SWAP] └─rhel_rdma--virt--03-home 253:2 0 410.8G 0 lvm /home