Project: GraphScope (App Definition and Development / Database)
File: performance_and_benchmark.md

We evaluated performance of GraphScope with the benchmarks provided by . Specifically, we evaluated Graph Analytical Engine (GAE) with , which is an industrial-grade benchmark that enables the objective comparison of graph analysis platforms. We also evaluated Graph Interactive Engine (GIE) with LDBC Social Network Benchmark, which focuses on graph database management systems and includes two workloads for interactive transactional queries and analytical queries. We compared GraphScope's performance to the state-of-the-art systems including PowerGraph, GeminiGraph, Plato, Neo4j, etc. and found GraphScope achieves high performance in most cases. We evaluated performance of Graph Analytical Engine (GAE, in short in the following) with . In addition to the ease of programming, we find that GAE achieves high performance comparably to the state-of-the-art systems. The experiments were conducted on 4 instances of on , each with 32 threads, over LDBC XL-size datasets. Instances are imaged with . The datasets can be downloaded from . We compared GAE with (commit a038f97 ) (commit 170e7d3 ) and (commit 21009d6). Each system is built with GCC(v4.8.5) and MPICH(v3.1). To make the comparisons fair, GAE was built with HUGE_PAGES and jemalloc disabled. We made minor changes on their code and datasets: Turned on `-O3` optimization for all three systems. Added timing stubs for Plato. Replaced the random-pick logic with the deterministic logic from LDBC for the CDLP in Plato. Changed the weight type of SSSP from `float` to `double` in GeminiGraph. Changed the load strategy from `loaddirected` to `loadundirectedfromdirected` for PageRank in GeminiGraph Reformat the datasets to adapt Plato and GeminiGraph's formats (e.g., 0-based continuous vertex ids) The results are reported below. The numbers in the table represent the evaluation time in seconds. The best results are marked in bold. | Algorithm | Dataset | PowerGraph | GeminiGraph | Plato | GAE | |--|-||-|-|| | SSSP | datagen-9_0-fb | 5.08 | 0.62 | N/A | 0.42 | | | datagen-9_1-fb | 5.30 | 0.78 | N/A | 0.56 | | | datagen-9_2-zf | 41.19 | 3.75 | N/A | 1.48 | | WCC | datagen-9_0-fb | 14.14 | 0.88 | 2.60 | 0.41 | | | datagen-9_1-fb | 18.61 | 1.17 | 3.07 | 0.50 | | | datagen-9_2-zf | 176.87 | 6.26 | 25.49 | 1.32 | | | graph500-26 | 13.71 | 1.60 | 4.79 | 0.71 | | | com-friendster | 44.20 | 3.97 | 7.80 | 1.97 | | BFS | datagen-9_0-fb | 3.90 | 0.24 | 0.59 | 0.07 | | | datagen-9_1-fb | 4.30 | 0.28 | 0.71 | 0.13 | | | datagen-9_2-zf | 39.11 | 1.97 | 10.37 | 1.16 | | | graph500-26 | 4.86 | 0.53 | 1.56 | 0.20 | | | com-friendster | 12.80 | 1.09 | 2.67 | 0.74 | | PageRank | datagen-9_0-fb | 22.57 | X | X | 1.40 | | | datagen-9_1-fb | 28.38 | X | X | 1.73 | | | datagen-9_2-zf | 126.98 | X | X | 3.83 | | | graph500-26 | 28.66 | X | X | 2.42 | | | com-friendster | 57.10 | X | X | 6.04 | | CDLP | datagen-9_0-fb | 1695.73 | N/A | 16.30 |

| | | datagen-9_1-fb | 2742.47 | N/A | 21.35 | 10.40 | | | datagen-9_2-zf | > 3600 | N/A | 34.85 | 19.48 | | | graph500-26 | 425.55 | N/A | 12.86 | 7.59 | | | com-friendster | > 3600 | N/A | 36.87 | 19.10 | | LCC | datagen-9_0-fb | 521.26 | N/A | N/A | 14.51 | | | datagen-9_1-fb | 600.32 | N/A | N/A | 18.35 | | | datagen-9_2-zf | 296.18 | N/A | N/A | 8.98 | | | graph500-26 | 1859.86 | N/A | N/A | 201.20 | | | com-friendster | 842.68 | N/A | N/A | 61.44 | We used default code provided by the competitor systems when it is available. N/A indicate that the system didn't provide the application. And X indicates the results produced are not consistent with the verified results provided by LDBC. The inconsistences of PageRank come from different settings on convergence conditions. To give a comprehensive comparison, we made our best efforts to revise our application(), making them output the same results as competitor systems. The performance results are shown as below. | Algorithm | Dataset | GeminiGraph | Plato | GAE | |--|-|-|-|| | PageRank | datagen-9_0-fb | 2.21 | 4.65 | 1.39 | | | datagen-9_1-fb | 2.72 | 5.38 | 1.73 | | | datagen-9_2-zf | 7.84 | 36.11 | 3.63 | | | graph500-26 | 4.75 | 12.25 | 2.34 | | | com-friendster | 8.19 | 15.82 | 5.84 | We compare the GAE GPU version with (commit 0c9a96, tag:v0.5.1) and (commit 5ce160). Each system is built with GCC(v7.3.0), OpenMPI(v4.1.1), and CUDA(v10.0). The experiments were conducted on GPU instances of on , equipped with 8x 16GB NVIDIA-V100 GPU, and 256 GB host memory. Instances are imaged with . Since the device memory of GPUs is limited, we conducted the comparison on several smaller graphs from the paper of Gunrock and Groute. The results are reported below. The numbers in the table represent the evaluation time in milliseconds. The best results are marked in bold. | Algorithm | Dataset | Gunrock | Groute | GAE-GPU | |-||--|-|-| | SSSP | soc-LiveJournal1 | 89.9345 | 215.560 | 32.3029| | | soc-twitter-2010 | 129.454 | 389.155 | 88.0549| | | soc-sinaweibo | 210.168 | 1649.647 | 172.03| | | soc-orkut | 105.121 | 193.658 | 54.3201| | WCC | soc-LiveJournal1 | 50.7677 | 12.137 | 10.2139| | | soc-twitter-2010 | 319.261 | 69.235 | 66.334| | | soc-sinaweibo | 567.09 | 283.073 | 68.6619| | | soc-orkut | 45.777 | 7.150| 8.87299| | BFS | soc-LiveJournal1 | 28.162 | 51.000 | 5.60403| | | soc-twitter-2010 | 45.9676 | 141.347 | 27.7772| | | soc-sinaweibo | 55.0118 | 168.306 | 34.8618| | | soc-orkut | 41.9134 | 37.467 | 6.60801| | PageRank | soc-LiveJournal1 | 39.8666 | 3501.870 | 24.6451| | | soc-twitter-2010 | 198.829 | 5036.229 | 127.024| | | soc-sinaweibo | 483.178 | 5593.807 | 228.367| | | soc-orkut | 57.6946 | N/A | 49.2339| We will release a public image containing the script, together with all the systems and datasets on AlibabaCloud and AWS soon. TBF