bug fixes to run on spark cluster