SubdomRobots checkpoint needed as I'll end my day very soon.
Rough draft of the SubdomRobots feature. Lots of test fails at this
point.
Fixed bug with robotstxt.Test. SubdomRobots now passes existing tests.
Merge branch 'master' into SubdomRobots
Added tests to verify SubdomRobots is working correctly.
Clean up of docs for SubdomRobots patch.
Merge pull request #86 from iParadigms/SubdomRobots

Subdom robots
Merge branch 'master' into ContentSize

Conflicts:
	fetcher.go
	test/fetcher_test.go