## 5 Experimental Evaluation
### Experimental Results
If the server finds a request from a client, it responds to the request. Each client is mapped to one dedicated core to avoid context switching. Each client sends 8 bytes message request with 512 unsignaled operations. Figure 12 shows the performance by increasing number of clients. The degradation of SEND and WRITE throughput happens because the SENDs and WRITEs are unsignaled, i.e., client processes get no indication of operation completion. This leads the client processes overwhelming with too many outstanding operations, causing cache misses inside the HCA. As can be seen, WRITE and READ outperform the other operations. Furthermore, with few clients the WRITE outperforms the READ operation. Surprisingly, SEND underperforms socket communication with higher number of clients. It should be noted that socket is implemented in non-blocking mode in this experiment. Furthermore,  
Figure 8: Inline and connection type impact on the SEND performance. Figure 9: Inline and connection type impact on the WRITE performance. only one machine is adopted to deploy the client processes. However, increasing the number of client machines can alter the performance because the overhead of maintaining the QP states are distributed among different machines [14]. The communication paradigms are evaluated and discussed in the following. Figures 13 and 14 show the performance and the latency of communication paradigms, respectively. According to the experiments reported in Figure 12, WRITE operation has the best performance, thus it was expected that _WRITE-WRITE_ outperforms the other communication paradigms. Surprisingly, the _WRITE-READ_ has much larger (i.e., 2.4 \(\times\)) throughput than _WRITE-WRITE_. The reason is that in _WRITE-WRITE_ communication, a client sends its request and polls the response area, and increasing the number of clients overwhelms the server to process requests and replies to them by WRITE operation. However, _WRITE-READ_ paradigm lightens the server burden, since server replies to requests by writing to local memory. Thus, it can process more requests and reduce the response time of clients,  
Figure 11: Performance comparison on reliable connections. Figure 10: Performance comparison on unreliable connections. consequently achieving a better throughput. Furthermore, Figure 14 demonstrates that the latency of _WRITE-READ_ paradigm is the lowest among the other paradigms. Unlike _WRITE-READ_, _READ-WRITE_ overloads server by reading requests through READ and replies by WRITE. Thus, the _READ-WRITE_ underperforms the _WRITE-READ_. The _READ-READ_ has better throughput than the _READ-WRITE_ because server only performs a READ operation comparing to a READ and a WRITE in _READ-WRITE_ paradigm. As demonstrated in Figure 13, _SEND-SEND_ and _SOCKET-SOCKET_ perform poorly due to their heavy operations, which was expected based on the results in Figure 12.  
_WRITE-READ_ is the only paradigm in the experiment that scales well by increasing the number of clients, however the primary drawback of this approach is that the client requires more effort in exchanging request-response. Thus, it is a good option for environments where clients are placed with other processes on the same machine because the amount of CPU that must be allocated to client  
Figure 12: Scaling different operations. Figure 13: Throughput of communication paradigms. processes is significantly high. _READ-READ_ is a good option to act fairly on both client and server sides. However, it performs 3.9 times slower than _WRITE-READ_. Moreover, _READ-READ_ incurs huge network traffic to poll remote host.