Added robotparser.
Merge pull request #87 from achesak/master

Added robotparser.