Fetches robots.txt unconditionally. Adds --no-strong-robots option.

Closes chfoo/wpull#58.
Implicitly span hosts on redirects.

Adds --no-strong-redirects. Adds reason to accept_url()'s info dict.
WebProcessorSession._should_fetch() now returns a tuple.

Closes chfoo/wpull#29.
Updates terse_options.rst.
Merge branch 'develop' of github.com:chfoo/wpull into develop

Conflicts:
	doc/changelog.rst