Add non-HTML URLs to the TTLHashSet

To prevent us from retrying non-HTML URLs each time we encounter them,
ensure that they are marked in the TTLHashSet (backed by Redis) as
'already crawled', meaning they won't be retried for as a long as the
TTLHashSet's expiry time (currently 48 hours).
Merge pull request #55 from alphagov/76505094-add-non-html-to-ttlhashset

[#76505094] Add non-HTML URLs to the TTLHashSet