Improve perf by removing a loop that does IO
Add support for reading from sitemap.xml
Add support for an external spidering skip list