Changed duplicate url behaviour.

1. Filter duplicates on save-level, to allow tests like
   "depth" to work in the save rules.

2. Filter duplicates before even adding them to the queue, to
   make the output a bit less verbose.

The console now lists the added links as +added/total.

Also remove some related code duplication, make _known_urls a set
Do not stumble over invalid expires headers.