added robots.txt option
Merge pull request #45 from idleberg/2.0-dev

added robots.txt option