Fixed bug on keeping the URL list, check duplicates against current URL,
crawled and not crawled URLs.

  Fixes #47