something inspired by an R package - rewritten to scale to big data.