some changes to account for spark's removal