BBC-News/wraith — pull request #149 by pmuir (2014-06-29 18:19:25+00:00)
Title: Misc fixes for spider.rb

* Use relative path to site base as label, substituting _ for / (resolves #140)
* Remove while replacing it with a ruby one-liner
* chomp trailing slashes, as foo/ is the same as foo
* downcase paths, as most filesystems and webservers ignore case

The last one might not be a good change, as some webservers may allow case to affect the URL loaded, not sure.

Also happy to rewrite without the rubyfication of the loop, if you prefer ;-)