Research Statement - Tera-Scale Graph Analysis

My vision is a big data analytics system which finds useful patterns and anomalies in graphs. Graphs are ubiquitous: computer networks, social networks, mobile call networks, protein regulation networks, and the World Wide Web, to name a few. The large volume of available data, the low cost of storage and the stunning success of online social networks and Web2.0 applications all lead to graphs of unprecedented size. Mining large graphs help us discover patterns and anomalies which can be useful for applications ranging from cyber-security (computer networks), fraud-detection (phone companies), and spammer detection (social networks). Typical graph mining algorithms silently assume that the graph fits in the memory of a typical workstation, or at least on a single disk; the above graphs violate these assumptions, spanning multiple Giga-bytes, and heading to Teraand Peta-bytes of data. As a consequence, the vast majority of large graphs has remained untouched. My research aims to unleash the potential by making extremely scalable graph mining algorithms on distributed platforms. Toward this goal, I have researched on algorithms for scalable graph mining and machine learning analysis. These algorithms are designed to work efficiently on distributed processing platforms including MapReduce and its open source version Hadoop, and analyze patterns and anomalies on very large graphs with more than billions of nodes and edges. Specifically, I have worked on the following algorithms. Structure Analysis. How to find patterns and anomalies in large, real world graphs? Extracting structural features (e.g. radius, diameter, closeness [5], connected components [3], and PageRank scores) is crucial for the discovery of patterns and anomalies in graphs. Previous researches focused on small graphs which fit in a memory or discs of a single machine, but they do not scale to very large graphs with billions of nodes and edges. My goal is to scale up the structure analysis algorithms for very large graphs, and to make the algorithms as general as possible so that we do not reinvent the wheel for similar problems. My approach is to develop an approximation algorithm to extract features much efficiently with a high accuracy. I developed Hadi algorithm [7, 8], an approximation algorithm for computing radius and diameter, which provides more than 97% of accuracy but is much efficient than its exact counterpart. Furthermore, I developed an important primitive called GIM-V (Generalized Iterative Matrix-Vector multiplication) [9, 10] to unify many seemingly different feature extraction algorithms. The advantage of the generalization is that an optimization helps boost the performance of many seemingly different algorithms. For example, my block encoding and clustering idea [9, 10] improved the performance by more than 5× compared to the naive algorithm. Figure 1 (a,b) show the GIM-V in action to discover patterns and anomalies in real graphs. Figure 1 (a) displays that the diameter of a very large web graph with billions of nodes and edges is very small, while there exist few anomalous ‘whiskers’ with large radii. This is the first result of the small world phenomenon on the Web. Figure 1 (b) shows the detection of two