[#76505094] Add non-HTML URLs to the TTLHashSet

To prevent us from retrying non-HTML URLs each time we encounter them,
ensure that they are marked in the TTLHashSet (backed by Redis) as
'already crawled', meaning they won't be retried for as a long as the
TTLHashSet's expiry time (currently 48 hours).