final working version for spark