Keep track of URLs we've already downloaded.

Yes, wget does this, but seems to only do so for each URL in a URL list.
Because our URLs are guaranteed to be related (i.e. they're all from the
same site) we can save ourselves a lot of work by keeping our own "have
we seen this" index.

URLs are MD5-hashed because I think we can save ourselves a fair bit of
memory by doing that vs. storing URLs as-is (which can get a bit long).
We'll see if my hunch is right.