more progress and refactor, in prep for doing uploads. Fix some other
unhandled error cases too