add scraper code that respects robots.txt