Whitespace
Add option to respect robots.txt disallow directives
Call robots_allowed with $this-> instead of self::
Merge pull request #12 from ericras/robots

Respect robots.txt