Fetch robots.txt with same scheme as url being examined

https://github.com/saltybeagle/Spider/pull/12#commitcomment-3030310