Added robots.txt parsing, and no robots.txt support

Parsing of robots.txt is done throught github.com/temoto/robotstxt-go.