1. Added back the configurable hdfs partition
2. updated the documentation