Fetches robots.txt unconditionally. Adds --no-strong-robots option.

Closes chfoo/wpull#58.
Implicitly span hosts on redirects.

Adds --no-strong-redirects. Adds reason to accept_url()'s info dict.
WebProcessorSession._should_fetch() now returns a tuple.

Closes chfoo/wpull#29.
Updates terse_options.rst.
Merge branch 'develop' of github.com:chfoo/wpull into develop

Conflicts:
	doc/changelog.rst
changelog.rst: Fixes previous merge on 2013 typos
document.py: Fixes wrong arguments passed.

Adds another shift_jis sample and unit test.
writer.py: Fixes files saved into wrong path (extra directory).

Closes chfoo/wpull#60.
url.py: Adds relaxed safe chars for URL path normalization.

Closes chfoo/wpull#68.
changelog: Updates latest to 0.24.
Merge commit '658b4c1e2418e814feb33d96489dcb239de0bb9a'
Bumps version to 0.24.