Misc fixes for spider.rb

* Use relative path to site base as label, substituting _ for / (resolves #140)
* Remove while replacing it with a ruby one-liner
* chomp trailing slashes, as foo/ is the same as foo
* downcase paths, as most filesystems and webservers ignore case
Add support for an external spidering skip list