added ability to download files during crawl
modify outbound request to ensure no gzip or defalt is returned
added test for gzip