Fetch robots.txt with same scheme as url being examined
Merge pull request #15 from ericras/robots

Fetch robots.txt with same scheme as url being examined