Project: Apache bRPC (Orchestration & Management / Remote Procedure Call)
File: rdma.md

Since RDMA requires driver and hardware support, only the build on linux is verified. With config_brpc ```bash sh config_brpc.sh --with-rdma --headers="/usr/include" --libs="/usr/lib64 /usr/bin" make cd example/rdma_performance # example for rdma make ``` With cmake ```bash mkdir bld && cd bld && cmake -DWITH_RDMA=ON .. make cd example/rdma_performance # example for rdma mkdir bld && cd bld && cmake .. make ``` With bazel: ```bash bazel build --define=BRPCWITHRDMA=true example:rdmaperformanceserver bazel build --define=BRPCWITHRDMA=true example:rdmaperformanceclient ``` RDMA does not use socket API like TCP. However, the brpc::Socket class is still used. If a user sets ChannelOptions.userdma or ServerOptions.userdma to true, the Socket class created has RdmaEndpoint (see src/brpc/rdma/rdma_endpoint.cpp). When RDMA is enabled, the data which need to transmit will be posted to RDMA QP with verbs API, not written to TCP fd. For data receiving, RdmaEndpoint will get completions from RDMA CQ with verbs API (the event will be generated from a dedicated fd and be added into EventDispatcher, the handling function is RdmaEndpoint::PollCq) before parsing RPC messages with InputMessenger. brpc uses RDMA RC mode. Every RdmaEndpoint has its own QP. Before establishing RDMA connection, a TCP connection is necessary to exchange some information such as GID and QPN. We call this procedure handshake. Since handshake needs TCP connection, the TCP fd in the corresponding Socket is still valid. The handshake procedure is completed in the AppConnect way in brpc. The TCP connection will keep in EST state but not be used for data transmission after RDMA connection is established. Once the TCP connection is closed, the corresponding RDMA connection will be set error. The first key feature in RdmaEndpoint data transmission is zero copy. All data which need to transmit is in the Blocks of IOBuf. Thus all the Blocks need to be released after the remote side completes the receiving. The reference of these Blocks are stored in RdmaEndpoint::sbuf. In order to realize receiving zero copy, the receive side must post receive buffers in Blocks of IOBuf, which are stored in RdmaEndpoint::rbuf. Note that all the Blocks posted in the receive side has a fixed size (recvblocksize). The transmit side can only send message smaller than that. Otherwise the receive side cannot receive data successfully. The second key feature in RdmaEndpoint data transmission is sliding window flow control. The flow control is to avoid fast transmit side overwhelming slow receive side. TCP has similar mechanism in kernel TCP stack. RdmaEndpoint implements this mechanism with explicit ACKs from receive side. to reduce the overhead of ACKs, the ACK number can be piggybacked in ordinary data message as immediate data. The third key feature in RdmaEndpoint data transmission is event suppression. The size of every message is limited to recvblocksize (default is

If every message will generate an event, the performance will be very poor, even worse than TCP (TCP has GSO/GRO). Therefore, RdmaEndpoint set solicited flag for every message according to data size, window and ACKS. The flag can control whether to generate an event in remove side or not. All the memory used for data transmission in RDMA must be registered, which is very inefficient. Generally, a memory pool is employed to avoid frequent memory registration. In fact, brpc uses IOBuf for data transmission. In order to realize total zerocopy and compatibility with IOBuf, the memory used by IOBuf is taken over by the RDMA memory pool (see src/brpc/rdma/block_pool.cpp). Since IOBuf buffer cannot be controlled by user directly, the total memory consumption in IOBuf should be carefully managed. It is suggested that the application registers enough memory at one time according to its requirement. The application can manage memory by itself and send data with IOBuf::appenduserdatawithmeta. In this case, the application should register memory by itself with rdma::RegisterMemoryForRdma (see src/brpc/rdma/rdmahelper.h). Note that RegisterMemoryForRdma returns the lkey for registered memory. Please provide this lkey with data together when calling appenduserdatawith_meta. RDMA is hardware-related. It has some different concepts such as device, port, GID, LID, MaxSge and so on. These parameters can be read from NICs at initialization, and brpc will make the default choice (see src/brpc/rdma/rdma_helper.cpp). Sometimes the default choice is not the expectation, then it can be changed in the flag way. Configurable parameters: rdmatraceverbose: to print RDMA connection information in logdefault is false rdmarecvzerocopy: enable zero copy in receive sidedefault is true rdmazerocopymin_size: the min message size for receive zero copy (in Byte)default is 512 rdmarecvblock_type: the block type used for receiving, can be default(8KB)/large(64KB)/huge(2MB)default is default rdmapreparedqp_size: the size of QPs created at the beginning of the applicationdefault is 128 rdmapreparedqp_cnt: the number of QPs created at the beginning of the applicationdefault is 1024 rdmamaxsge: the max length of sglist, default is 0, which is the max length allowed by the device rdmasqsize: the size of SQdefault is 128 rdmarqsize: the size of RQdefault is 128 rdmacqepoll_once: the number of CQE pooled from CQ oncedefault is 32 rdmagidindex: the index of local GID table useddefault is -1which is the maximum GID index rdma_port: the port number useddefault is 1 rdma_device: the IB device namedefault is emptywhich is the first active device rdmamemorypoolinitialsize_mb: the initial region size of RDMA memory pool (in MB)default is 1024 rdmamemorypoolincreasesize_mb: the step increase region size of RDMA memory pool (in MB)default is 1024 rdmamemorypoolmaxregions: the max number of regions in RDMA memory pooldefault is 16 rdmamemorypool_buckets: the number of buckets for avoiding mutex contention in RDMA memory pooldefault is 4 rdmamemorypooltlscache_num: the number of thread local cached blocks in RDMA memory pooldefault is 128