Master now uses database to store crawled URLs.
Merge pull request #44 from stbnps/master

Master now uses database to store crawled URLs.