req module pretends to be real browser
sitemap corrects gzip using two bytes header
robotstxt processes data correctly