more pre-processing steps
ported to work with HDFS