Code cleanup
adding support for uniq-only overflow queue
Using payload field as key instead of creating a new index
Discard items already visited
code cleanup
md file added
Doc updated
include_query_string_in_uuid is now correctly propagated to the storage engine
Fix if statement
It now supports redirects
typo fixed
s3 storage is added
You can now specify a block to extract urls from a page
regenerating stubs
A Redis connection can be estabilished by passing a block
Graceful exiting strategy added