From Data to Knowledge - 203 - Steve Plimpton
Channel: ckleinastro
Date: 2012-05-08

Steve Plimpton: "PHISH framework for streaming graph algorithms". A video from the UC Berkeley Conference: From Data to Knowledge: Machine-Learning with Real-time and Streaming Applications (May 7-11, 2012).

Abstract
Steve Plimpton (Sandia National Labs, Computational Sciences)
We've developed a small, portable C++ library called PHISH (Parallel Harness for Informatic Stream Hashing) which orchestrates the passing of datums between independent processes (minnows) as they compute on a stream of data in parallel. The library has a C API and a Python wrapper, so minnows can be written in any language (C, C++, Fortran, Python). PHISH can be run on top of message-passing (MPI) or sockets (zeroMQ), which means a streaming computation can be launched on a multi-core desktop, a traditional supercomputer, or a geographically disperse network. 

We've been using PHISH to experiement with streaming MapReduce operations and graph algorithms, such as connected component finding and sub-graph isomorphism matching. I'll give a few PHISHy details, describe the streaming graph algorithms at a high level, and highlight MPI vs socket performance.