removed non spark implementation functions