robotex gem included
robots.txt support added
code cleanup
sample added
Merge pull request #30 from taganaka/robotstxt

Support for robots.txt