hdfs cleanup:
    - dataframes made optional
    - streams closed when job is deleted
    - additional checks on hdfs apis
    - optimized single byte read from hdfs stream