Comment, remove unused arguments.
Add limited robots support.

Works via a robots test, but integration needs improvement.
Remove completed TODOs.
When a url is invalid, be clear as to which one.
Support local input files.
Support cookie persistence, configurability.