## 4 Communications
### Optimization Considerations  
Since RDMA is completely implemented in HCA with limited computing and memory resources, optimization techniques can significantly impact on the performance. In this section, possible optimizations are briefly described.  
**Data inlining:** WQEs are built from different segments depending on the requested operation. One important segment is _Data_ which points to the memory that contains the payload. WQEs can optionally contain data known as _inline data_. Inline data will save a memory read access for gathering  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**METHOD** & **CLIENT OVERHEAD** & **Server OVERHEAD** & **NEWORK TRAffic** & **COMMUNICATION** & **CONNECTION** \\ \hline _WRITE-WRITE_ & high & high & low & a/synchronous & RC/UC \\ _READ-WRITE_ & low & high & high & synchronous & RC \\ _WRITE-READ_ & high & low & high & synchronous & RC \\ _READ-READ_ & fair & fair & high & synchronous & RC \\ _SEND-SEND_ & high & high & low & a/synchronous & RC/UC \\ _SOCKET-SOCKET_ & high & high & low & a/synchronous & RC/UC \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparing communication paradigms.  
Figure 4: Communication paradigms.  
the payload. The WQE can contain single or multiple memory pointers as well as inline data, or any combination of these. Although inlining data can eliminate the overhead of memory access, it imposes limitation on the payload size. Anyhow, it is an efficient optimization for small payload sizes.  
**Payload size:** Payload size plays a crucial role in the performance. It can increase the number of communications between CPU, memory, and HCA, consequently degrading the performance. For example, in case of 64 bytes cache line size if payload is increased from 64 to 65 bytes, two can lines must be accessed in transferring the payload to HCA. Furthermore, performance can be affected by the host bus (i.e., _Peripheral Component Interconnect Express (PCIe)_) parameters. For example, _Maximum Payload Size (MPS)_ defines the maximum transfer size in each PCIe communication. If MPS is low, it can increase the number of communications between HCA and memory [19].  
**Transport types:** RMDA supports three different connection types: _Reliable Connected (RC)_, _Unreliable Connected (UC)_, and _Unreliable Datagram (UD)_. UD is an unreliable and unconnected connection which does not guarantee the delivery as well as the order of the messages. The connection types not only affect on the network communication traffic but also on the amount of consumed resources in HCA [14]. For example, UD connections consume much less resources comparing to RC and UC. The UC/UD may have higher performance comparing to RC at the expense of losing the reliability. However, zero packet loss was experienced in transferring 50 PB of data over an unreliable connection. [PERSON] _et al._[20]. Furthermore, it should be noted that each connection type does not support all RDMA operations. Table 2 shows the supported operations by each connection type.  
**Unsignaled work request:** When a work request is processed, a work completion element is generated by HCA. Generating this element incurs overhead on the application to collect it. RDMA allows to send a work request without receiving the completion element. However, this approach cannot be adopted continuously due to the resource depletion. Thus, a completion work request must be requested periodically to release the taken resources. Furthermore, RDMA allows reading multiple completion elements at once.