config: accidentally added two discover_hosts keys
SubdomRobots checkpoint needed as I'll end my day very soon.
Upping fetcher test timeout, still seeing failures
Upped cassandra timeout to help travis ci
Merge pull request #85 from iParadigms/extra-config

config: accidentally added two discover_hosts keys
Rough draft of the SubdomRobots feature. Lots of test fails at this
point.
Fixed bug with robotstxt.Test. SubdomRobots now passes existing tests.
Merge branch 'master' into SubdomRobots
Added tests to verify SubdomRobots is working correctly.
Clean up of docs for SubdomRobots patch.
dispatcher: lower consistency on link selection

Lowered selection of all links for a domain to One. See diff for
explanation.
Merge pull request #87 from iParadigms/dispatcher-select-consistency-one

dispatcher: lower consistency on link selection
Merge pull request #86 from iParadigms/SubdomRobots

Subdom robots
Merge branch 'master' into NormalizeUrl