refactored for speed.  Now runs in ~35 sec on full data set