# Locality-Sensitive Hashing for Information Retrieval System on Multiple GPGPU Devices
## 6 Results and Comparison  
### Performance of S-PLSH on Multiple GPGPU Devices  
First, we carried out experiments of Multi-stage search and S-PLSH, with different numbers of GPGPU devices. As shown in Figure 10, when dealing with the same database that has the same number of queries, using multiple data clusters (Blind search) helps to reduce the search time by parallel processing on multiple processes. With a higher number of slaves, we can achieve a lower memory size of the database for each data cluster. The S-PLSH algorithm resulted in a shorter search time than Multi-stage search and Blind search, as used in [16], by skipping numerous probes. However, S-PLSH took time to share information among data clusters. As shown in Figure 11, the recall of S-PLSH was comparable with recall of Multi-stage search and Blind search. As S-PLSH has the potential to skip the probes that \begin{table}
\begin{tabular}{l l} \hline \hline
**CPU** & **Intel Xeon Gold 6130 32 Cores** \\ \hline CPU Clocks & 2.1 GHz \\ Main Memory & 64 GB \\ Bandwidth (internal node) & 3088 MB/sec \\ Bandwidth (between nodes) & 1202 MB/sec \\ GPGPU & Tesla P100 (\(\times\)2) \\ GPGPU Clocks & 1328 MHz \\ GPGPU Memory & 16 GB (\(\times\)2) (GDDR5) \\ CUDA version & 9.1 \\ Recall reduction (4 nodes vs. 1 node) & 0.083\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Specifications of each testing node (8 nodes total). have true approximate neighbors with the same similarity as the candidates on other data clusters, the accuracy of S-PLSH slightly decreased when the number of slaves increased. In addition, in the case of the system with multiple slaves, the search speed can be affected by the bandwidth. Figure 10 shows the differences in slave search times on the same and different nodes. This shows, in the case of two slaves, the overhead of searching on different nodes (1 GPGPU/node) was about 5-7%. In addition, the distance computation between items takes most of the time in the searching process. When using HiFP2.0 audio fingerprint, taking bit-to-bit XOR operations for the sequences of 4096 bits takes more GPGPU clock cycles than the hashing computation of the vector of 4096 bits. The impact of different synchronization strategies is shown in Figure 10, where S-PLSH* is the S-PLSH search using MPI messages broadcasted among slaves without using the Master as an intermediary. With a small number of slaves (e.g., 2-8), S-PLSH* showed better performance than S-PLSH, due to the fast transfer. It is clear that, with 10,000 HiFP2.0 features on 16 slaves, S-PLSH sent a total of 2.5 MB/(32 messages) of data for synchronization while S-PLSH* needed 18.75 MB/(240 messages). These amounts of memory were small, compared to the network bandwidth in the test computer. This indicates that the results are acceptable. There was a trade-off between bandwidth and performance in S-PLSH's strategies. For this reason, S-PLSH* is not recommended for use in a search system with a higher number of slaves or with small bandwidth. The synchronization performance results are shown in Figure 12. The linear effect of the number of items is clearly shown in this chart. As the message size is smaller with a higher number of slaves, our system was able to process more queries. To send large messages, we could divide them into small messages and send them multiple times. However, the sending time will sharply increase, due to the overhead of sending and receiving packages and management cost. The networks between processes on the same node were always faster than networks on different nodes, which made the synchronization time of eight slaves on two nodes higher than that of eight slaves on the same node. With 16 slaves, we could process the result synchronization process of 1 million queries in 5 s. However, in real-world cases, we do not need to process that many queries at once. Instead, we can split them up to ensure the maximum search time for all queries.