Cleanup
- modified robots.txt
Merge pull request #1 from afribase/allow.crawlers.to.scan.website

Cleanup