scrapy/scrapy — Issue #89: Support for gzipped sitemaps in SitemapSpider
Author: rosner
Created: 2012-02-12 12:38:03+00:00
State: open

Since a lot of sites use gzipped sitemaps defined in robots.txt it would be great if the SitemapSpider would support those when a robots.txt is given in the sitemap_url property.