crawler: now with error handling

get_url is called within a try block; to avoid issues such as unhandled exceptions leaving Postgres in an error state, transaction.rollback() is called after every request
Switch to logging

Slowly starting the process of converting print statements to logging calls. Still todo: initializing logging for people whose projects don't already do that.
crawler: better test environment

Disable settings.DEBUG before crawling for performance
Use test client's setup/teardown to avoid things like email being sent.
crawler: performance boost

BeautifulSoup will spend a huge amount of CPU time doing charset detection if you give it a non-unicode string
crawler: better logging support

Now a logger named "crawler" is enabled and configured by the management command for use by everything in the crawler.

Fixed a minor bug: CommandError was referenced but not imported

Cleaned some imports & dead code
Crawler: performance improvement for sanitize plugin
crawler: better link handling & logging
crawler: better validation & error handling

base.py now warns about non-200 HTTP status codes

sanitize.py now avoids validating non-HTML responses
Crawler: better redirect handling

The test client followed URLs; now it passes the final redirect_chain to
_parse_urls so relative URLs are joined correctly (i.e. if "/foo"
redirects to "/foo/" we want <a href="bar"> to become "/foo/bar", not
"/foobar").
Crawler: fixed URL parsing of non-HTML responses
crawler: better reporting of unmatched URL patterns
Crawler: consistent logging for uncrawled patterns
Crawler: non-zero return status if we encounter errors
TestMaker doc update
Merge branch 'master' of git://github.com/ericholscher/django-test-utils

Conflicts:
	test_utils/crawler/plugins/guppy_plugin.py
	test_utils/crawler/plugins/memory_plugin.py
Normalized whitespace for twill_runner.py
Merge branch 'master' of git://github.com/ericholscher/django-test-utils
Merge branch 'master' of git://github.com/ericholscher/django-test-utils

Conflicts:
	docs/source/testmaker.rst
Avoid logging warnings for non-error HTTP statuses
Merge branch 'master' of git://github.com/ericholscher/django-test-utils