Improve perf by removing a loop that does IO
Merge pull request #155 from pmuir/remove_redundant_code

Improve perf by removing a loop that does IO
Add support for reading from sitemap.xml
Merge pull request #152 from pmuir/sitemap_xml

Add support for reading from sitemap.xml
Add support for an external spidering skip list