follow robots.txt in issue #12
Merge pull request #14 from clarkfitzg/feature-craigslist

follow robots.txt in issue #12