traverse: minor refactoring and documentation
datastore: first pass implementation
datastore: added is_searched
datastore: adding logging
traverse: data now persisted in datastore
traverse: use urljoin result for filtering
datastore: fixed bug in storing results
datastore: simplified storage for now
datastore: not adding corpus to datastore
datastore: added expiring processing tags
traverse: tag URL that are being processed
traverse: replaced zmq/tornado

It made more sense to spawn multiple processes to process the data and use the
expiration functionality in redis to throttle calls to websites.
Merge branch 'working'