Misc fixes for spider.rb

* Use relative path to site base as label, substituting _ for / (resolves #140)
* Remove while replacing it with a ruby one-liner
* chomp trailing slashes, as foo/ is the same as foo
* downcase paths, as most filesystems and webservers ignore case
Merge pull request #149 from pmuir/140

Misc fixes for spider.rb
Improve perf by removing a loop that does IO
Merge pull request #155 from pmuir/remove_redundant_code

Improve perf by removing a loop that does IO
Add support for reading from sitemap.xml
Remove double write to spider_file
Add support for an external spidering skip list