improved some spec descriptions
redirect logging to a file, running specs uses a separate file
changed how non-scraped pages are stored (now as Page objects), added scaffolding for sitemap.xml report
added sitemap.xml generation, also attached to each node reason why it wasn't crawled
normalize urls by sorting parameters alphabetically
producing html report too
a couple of xml improvements