updating docs and refactoring
adding stuff for spark
more fixes