Fixed conditional reloading of job metadata and datafile contents

The caching of datafiles was not working at all because the call to
storing the collected data had ended up in the wrong place after the
code reorganization.

Job metadata will now be reloaded if the jobTypes data has changed.

Progress reports have been extended by a status line.

Data transmission can now be compressed. Furthermore, the meteredReader
will use an adaptive buffer_size to reduce impact. I also fixed my
O(N^2) readline() snafu.