some improvements, bug fixes, still need to process shuffle on disk though