added spark functions and configuration methods