accept only protocol http.
a store needs to have sorted set operations.
the sorted set is used to model the queue the crawler is working on.
using redis as a first implementation.