saltybeagle/Spider — pull request #15 by ericras (2013-04-17 14:50:51+00:00)
Title: Fetch robots.txt with same scheme as url being examined

#12 - https://github.com/saltybeagle/Spider/pull/12#commitcomment-3030310