Part 5: Playing Fair

Since we are writing a program that makes use of the public
infrastructure that is the Internet, it makes sense to play fair and
make our programs behave properly so that we can avoid clashes with the
webmasters, or even worse retaliation.

Webmasters may not want their site to be spidered by other computers,
and they have a way to say it clearly with the robots.txt file. Inside a
robots.txt file, which is a simple text file stating one or more rules,
a webmaster can describe if scraping is allowed at all, or otherwise
limit scraping to only some files and folders, or only to some specific
spiders.

Our plan to playing fair is to read the site’s robots file, and stay
well clear of the actual content if we are not allowed to read it. The
wisest thing to do here is to stand on the shoulders of giants, since
there are already other people who have done this in the past. And
luckily some of these folks have shared their efforts for everyone else
to use. Which means that we do not have to reinvent the wheel, and will
only write a few lines of code to read the robots.txt file and make our
program compliant.