NEXUS-5472: S3 Scraper with some other changes for all

Refactored Scraper to make it more capable to handle cases
like S3 is, where scraping of 404 pages is must too.