url queue is now independent and self managing. Duplicate domains are skipped.
Adding mongo-based persistence