Made some things configurable in principle.
Added robots.txt flag