SHDP-444 Backport of SHDP-447
SHDP-416 Change extension used for bzip2 and gzip

- Backport of SHDP-417, fixed merge conflicts.
SHDP-422 Backport of SHDP-405

- Mostly same as original commit expect minicluster
  based test not picked.
SHDP-422 Backport of SHDP-409

- AbstractDataStreamWriter has been enhanced for
  guarding against same leaseholder to try recreating
  same path.