Fetch robots.txt with same scheme as url being examined