Workaround for url parsing bug in PHP 5.3
robots.txt parsing
Add comments and code cleanup