re-generalized spark code