compartamentalized, added spark code