partial spark implementation.