saltybeagle/Spider — pull request #29 by mfairchild365 (2014-03-27 15:29:48+00:00)
Title: Trim whitespace from found uris

This issue was brought up via https://github.com/UNLSiteMaster/site_master/issues/57

The problem is that when we Tidy a page, it by default tries to fix all found URIs in two ways
* By escaping URIs http://tidy.sourceforge.net/docs/quickref.html#fix-uri
* By converting backslashes http://tidy.sourceforge.net/docs/quickref.html#fix-backslash

This causes an HREF value of ' http://www.google.com/' to be converted to '%20http://www.google.com', and according to http://www.w3.org/TR/html5/infrastructure.html#valid-url-potentially-surrounded-by-spaces that white space should be trimmed, not escaped.

For the purpose of logging, the spider should not alter these URIs at the parsing stage, and should instead fix them during the crawling stage if it needs to.