added blurb about robots.txt
Merge branch 'scrapy' of github.com:forrestglines/numerical_computing into forrest_scrapy
spelling pass
Merge pull request #2 from ayr0/forrest_scrapy

Robots.txt