Fetch robots.txt with same scheme as url being examined

#12 - https://github.com/saltybeagle/Spider/pull/12#commitcomment-3030310