Parse sitemaps to find links to visit.

* Given an robots.txt file, finds a sitemap and returns the
  links found inside.
Recursively download sitemaps.
Record discovered URLs.

After parsing a sitemap, save any of the URLs found in the database.