Added robots crawling (use 1 to enable it with httpcrawl)