added dupefilter, separated out words in url, title, body
write urls to file
considered only base url for filtering, not the query portion
added dump to gitignore
added redis power switch
added redis power switch
added url_id as a field
checking if a url in a pipeline already has its id assigned
changed start_url, domain
remap for mongodb
Few cosmetic changes. Nothing significant
added term frequency storage, more text cleaning
Removed comments. Added throttler. Using MongoDB to generate web.mtx and out_links.