Ignore robots.txt

Signed-off-by: Chris Lamb <lamby@debian.org>
Disable more things.

Signed-off-by: Chris Lamb <lamby@debian.org>
Add mcqueen