- Fixed issue with Ganymede - cobweb integration where crawling was
always attempting to read the sitemap regardless of the inputs
- Sitemap urls are now constructed correctly and follow redirects
properly in all cases
- Robots tweaks for input reading
- version bump