Scalable Algorithms for Mining Dynamic Graphs and Hypergraphs with Applications to Anomaly Detection.

RANSHOUS, STEPHEN MICHAEL. Scalable Algorithms for Mining Dynamic Graphs and Hypergraphs with Applications to Anomaly Detection. (Under the direction of Dr. Nagiza F. Samatova.) Graph data mining has become a ubiquitous tool for researchers and practitioners in numerous domains, including social sciences, financial markets, and computer security. In particular, mining dynamic graphs has gained substantial interest in the past decade, due to their robust expressiveness and natural ability to represent complex relationships that evolve over time. Traditional graph mining tasks, such as community detection, frequent subgraph mining, and anomaly detection, often operate on entire graph snapshots at a time, where a snapshot may consist of an entire day or week of activity. However, as the volume and velocity of data continues to increase, it is no longer feasible to analyze entire datasets at once, or even assume that the datasets will fit into memory. In this work we focus on developing efficient algorithms for mining dynamic graphs. We use anomaly detection as an exemplar task, but believe that our cross-cutting ideas and applications of efficient data structures apply much more broadly. We propose two changes for how anomaly detection is performed over large-scale dynamic graphs to cope with the growing constraints. First, we transition from the traditional approach of analyzing graph streams, where each object in the stream is a full graph, to analyzing graph edge streams. An immediate benefit of this transition is a smaller unit of analysis, as well as more fine-grained attribution for anomalies. Second, instead of storing exact information, we utilize approximation, heuristics, and probabilistic data structures for retaining the salient information from the stream. This results in dramatically reduced time and space complexities for our algorithms. In this dissertation we develop anomaly detection algorithms for increasingly complex graph edge stream models and show their effectiveness, both theoretically and empirically. In our first component, based on our extensive survey and gap analysis of the field, we begin with the simplest case, undirected graph edge streams. Key graph properties necessary for our anomaly detection algorithm are approximated from the stream using Count-Min (CM) sketches. Theoretical and empirical results show that the error of our approximations is minimal, while enabling constant time operations. Experiments on synthetic and real-world datasets show the effectiveness of our approach, yielding high precision and recall, as well as interesting case studies. In our second component, we plan to extend the streaming model from graph edges to hypergraph edges, or hyperedges. As hyperedges represent higher order relationships, not strictly pairwise, we must transition to a more flexible notion of similarity and frequency, thus we can no longer use CM sketches. Instead, we use MinHash and locality sensitive hashing to quickly and accurately estimate historical vertex co-occurrences. Experimental results show processing speedups between 33-750x compared to exact analysis, while maintaining comparable accuracy. Finally, our last component examines the potential for pattern based anomalies in dynamic directed hypergraphs (dirhypergraphs). We perform a case study using the Bitcoin network, and propose an edge-based pattern which we posit may represent money laundering or transfer. Patterns in the dirhypergraph are found heuristically, and are centered around “exchange vertices”. Experimental results show that our laundering pattern appears at a significantly low rate, and that exchange vertices have several unique characteristics, enabling accurate classification models to be constructed.