more progress on the pipeline, fixing a bug with reading data from the wrong directory and made some changes to enable the file formatter to be a first class member of the workflow