## 2 Background  
Although iWARP, RoCE, and InfiniBand protocols provide a unique set of operations, however adopting an appropriate protocol requires the awareness of advantages and drawbacks of each protocol. The iWARP [1] is a complex protocol published by _Internet Engineering Task Force_ (_IETF_) which only supports reliable connected transport (see Section 4.2) [12]. It was designed to convert TCP to RDMA semantics which limits the efficiency of iWARP products. On the contrary, RoCE is an Ethernet-based RDMA solution published by _InfiniBand Trade Association_ (ITA) supporting reliable and unreliable transports. InfiniBand is an advanced network protocol with low latency and high bandwidth commonly used in commodity servers.  
The _OpenFabrics Alliance_ (_OFA_) publishes and maintains user-level _Application Programming Interface_ (_API_) for the RDMA protocols (i.e., iWARP, RoCE, and InfiniBand) [13] along with useful utilities, called _OpenFabrics Enterprise Distribution_ (_OFED_).  
As Figure 1 shows, RDMA allows an application to queue up a series of requests to be executed by HCA. Queues are created in pairs, called _Queue Pair_ (_QP_), to send and receive operations. An application submits a _Work Queue Element_ (_WQE_) on the appropriate queue. Then, channel adapter executes WQEs in the FIFO order on the queue. Each WQE can work with multiple scatter/gather entries to read multiple memory buffers and to send them as one stream and write them to multiple memory buffers. When the channel adapter completes a WQE, a _Completion Queue Element_ (_CQE_) is enqueued on a _Completion Queue_ (_CQ_). RDMA provides various work requests consisting of different operations such as:  
* SEND/RECV sends/receives a message to/from a remote node
* _Fetch-And-Add_ (FAA) atomically returns and increments the value of a virtual memory location in the remote machine
* _Compare-and-Suzup_ (CAS) atomically compares the value of a virtual memory address with a specified value and if they are equal, a new value will be stored at the specified address
* READ/WRITE reads/writes a data from/to a remote node exploiting the _Direct memory Access_ (_DMA_) engine of the remote machine (i.e., bypassing the CPU and kernel)  
Figure 2 shows a possible RDMA communication between Machines A and B. Firstly, the user application issues a send request in the step 1 to communicate with the channel adapter through CPU in the step 3. Kernel space operation in the step 2 is used only for starting an RDMA connection, and there is not any operation or buffering when connection is set up. Furthermore, the existence of the fourth operation depends on the request type. If the data are inlined (see Section 4.3), the HCA does not need to perform an extra DMA operation to read the payload from the user-space memory. Afterwards, the request is enqueued in the send queue and is waiting for its turn to be processed by the _Network Processor_ (_NP_). When the message is received by Machine B, it can be performed on a memory address (step 4) without any reply to Machine A or with a reply to Machine A (step 5).  
Figure 1: InfiniBand Architecture.  
Figure 2: RDMA communication cutting view.