Update web-scraping.md

Adds more detail to robots.txt