# Interactive Graph Stream Analytics in Arkouda
## 8 Conclusions  
Interactive graph stream analysis is a challenging problem. In this paper, we present our solution to show how we can solve the problem in the open source framework Arkouda. The advantage of Arkouda lies in two aspects: high productivity and high performance. High productivity means that the end users can use a popular data science language such as Python to explore different graph streams. High performance means that the end users can break the limit of their laptop and personal computer's capabilities in memory and calculation to handle very large graphs in an interactive way. We design a double index data structure to support quick search from a given vertex to edges or from a given edge to vertices. This greatly reduces the total memory footprint for large sparse graph streams. Based on the proposed double index data structure, we develop an efficient multiple-partition stream sketch building method. Our sketch can support general stream query problems because our sketch uses a small graph to summarize a large graph. We define a shrinking factor to let users control the size of final sketch accurately. We exploit the ubiquitous normal and power law degree distributions of given graph streams to propose two regression models to estimate the results of the graph streams from their sketch partitions. Based on the double index data structure, we develop a multi-locale distributed triangle counting algorithm that can maintain load balancing and a high local \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Graph Name** & **Factor** & **Stream Time (s)** & **Exact Time (s)** & **Speedup** \\ \hline ca-GrQc & 16 & 2.02119 & 3.23613 & 1.60 \\ \hline ca-HepPh & 4 & 10.263 & 21.3077 & 2.08 \\ \hline ca-HepPh & 8 & 6.98969 & 21.3077 & 3.05 \\ \hline ca-HepPh & 16 & 5.45666 & 21.3077 & 3.90 \\ \hline ca-HepPh & 32 & 4.83524 & 21.3077 & 4.41 \\ \hline ca-HepPh & 64 & 4.3246 & 21.3077 & 4.93 \\ \hline ca-HepTh & 4 & 3.78554 & 5.37471 & 1.42 \\ \hline ca-HepTh & 8 & 2.92663 & 5.37471 & 1.84 \\ \hline ca-HepTh & 16 & 2.46299 & 5.37471 & 2.18 \\ \hline dblp & 16 & 22.0434 & 163.092 & 7.40 \\ \hline dblp & 32 & 17.8394 & 163.092 & 9.14 \\ \hline dblp & 64 & 15.4684 & 163.092 & 10.54 \\ \hline email-Enron & 4 & 16.133 & 63.1888 & 3.92 \\ \hline email-Enron & 8 & 10.859 & 63.1888 & 5.82 \\ \hline email-Enron & 16 & 8.04875 & 63.1888 & 7.85 \\ \hline lj & 512 & 409.672 & 6937.4 & 16.93 \\ \hline lj & 1024 & 393.518 & 6937.4 & 17.63 \\ \hline lj & 2048 & 397.583 & 6937.4 & 17.45 \\ \hline loc-brightkite\_edges & 4 & 14.1111 & 30.1892 & 2.14 \\ \hline loc-brightkite\_edges & 8 & 10.1597 & 30.1892 & 2.97 \\ \hline loc-brightkite\_edges & 16 & 8.09854 & 30.1892 & 3.73 \\ \hline youtube & 512 & 36.2397 & 951.094 & 26.24 \\ \hline youtube & 1024 & 34.9246 & 951.094 & 27.23 \\ \hline youtube & 2048 & 34.6978 & 951.094 & 27.41 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: \(Cont.\)data access ratio to improve the graph stream processing performance. All our methods have been integrated into the Arkouda framework and users can use a Jupyter Notebook to drive the interactive graph stream analytics pipeline easily. Experimental results on two kinds of large sparse graph streams--normal distribution and power law distribution--show that the proposed method can achieve very high approximate results with a mean error no larger than 4%. The average local access ratio is 74% and the graph stream processing speedup increases almost linearly with the shrinking factor. This work shows that Arkouda is a promising framework for supporting large scale interactive graph stream analytics. Of course, the reported work is just the first step in supporting a typical graph query, triangle counting. In future work, we will provide more graph stream algorithms based on our sketch methodology and will further optimize the performance of our algorithms in Arkouda. Conceptualization and methodology, Z.D. and D.A.B. ; software implementation, Z.D., O.A.R. and D.A.B. ; validation, J.P.; writing--original draft preparation, Z.D., O.A.R.