Nested sitemaps now working correctly
Sitemap support now works in the crawler, with an escape so that the
sitemap is the only thing crawled if use_sitemap option is
specified. Currently we do not continue if the sitemap is specified
Bug fixes with redirection of a requested url for a sitemap
Fixing a bug that caused the crawl to stop if no sitemap was found,
and/or no urls were in the sitemap. Also added exception raising to be
a bit more specific rather than a StandardError
version bump
Adding support to simple crawl for robots.
- Fixed issue with Ganymede - cobweb integration where crawling was
always attempting to read the sitemap regardless of the inputs
- Sitemap urls are now constructed correctly and follow redirects
properly in all cases
- Robots tweaks for input reading
- version bump
Fixing exception handling for Robots and sitemap parsing
GALAXY-722 - Merge with cobweb master