initial robots.txt handler added
added polite? option to config
politeness docs updated
Merge branch 'pr/5'
Whitespace fixes