Chapter 2. Configuring the rdma service With the Remote Direct Memory Access (RDMA) protocol, you can transfer data between the RDMA enabled systems over the network by using the main memory. The RDMA protocol provides low latency and high throughput. To manage supported network protocols and communication standards, you need to configure the rdma service. This configuration includes high speed network protocols such as RoCE and iWARP, and communication standards such as Soft-RoCE and Soft-iWARP. When Red Hat Enterprise Linux detects InfiniBand, iWARP, or RoCE devices and their configuration files residing at the /etc/rdma/modules/* directory, the udev device manager instructs systemd to start the rdma service. Configuration of modules in the /etc/rdma/modules/rdma.conf file remains persistent after reboot. You need to restart the rdma-load-modules@rdma.service configuration service to apply changes. Procedure Install the rdma-core package: Edit the /etc/rdma/modules/rdma.conf file and uncomment the modules that you want to enable: Restart the service to make the changes effective: Verification Install the libibverbs-utils and infiniband-diags packages: List the available InfiniBand devices: Display the information of the mlx4_1 device: Display the status of the mlx4_1 device: The ibping utility pings an InfiniBand address and runs as a client/server by configuring the parameters. Start server mode -S on port number -P with -C InfiniBand channel adapter (CA) name on the host: Start client mode, send some packets -c on port number -P by using -C InfiniBand channel adapter (CA) name with -L Local Identifier (LID) on the host:

Commands:
dnf install rdma-core
These modules are loaded by the system if any RDMA devices is installed iSCSI over RDMA client support ib_iser iSCSI over RDMA target support ib_isert SCSI RDMA Protocol target driver ib_srpt User access to RDMA verbs (supports libibverbs) ib_uverbs User access to RDMA connection management (supports librdmacm) rdma_ucm RDS over RDMA support rds_rdma NFS over RDMA client support xprtrdma NFS over RDMA server support svcrdma
systemctl restart <rdma-load-modules@rdma.service>
dnf install libibverbs-utils infiniband-diags
ibv_devices device node GUID ------ ---------------- mlx4_0 0002c903003178f0 mlx4_1 f4521403007bcba0
ibv_devinfo -d mlx4_1 hca_id: mlx4_1 transport: InfiniBand (0) fw_ver: 2.30.8000 node_guid: f452:1403:007b:cba0 sys_image_guid: f452:1403:007b:cba3 vendor_id: 0x02c9 vendor_part_id: 4099 hw_ver: 0x0 board_id: MT_1090120019 phys_port_cnt: 2 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 2048 (4) sm_lid: 2 port_lid: 2 port_lmc: 0x01 link_layer: InfiniBand port: 2 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
ibstat mlx4_1 CA 'mlx4_1' CA type: MT4099 Number of ports: 2 Firmware version: 2.30.8000 Hardware version: 0 Node GUID: 0xf4521403007bcba0 System image GUID: 0xf4521403007bcba3 Port 1: State: Active Physical state: LinkUp Rate: 56 Base lid: 2 LMC: 1 SM lid: 2 Capability mask: 0x0251486a Port GUID: 0xf4521403007bcba1 Link layer: InfiniBand Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x04010000 Port GUID: 0xf65214fffe7bcba2 Link layer: Ethernet
ibping -S -C mlx4_1 -P 1
ibping -c 50 -C mlx4_0 -P 1 -L 2