Crawler program

the following files manage the crawler, database and frontier.
currently it does not inspect hyperlinks
Revert "Crawler program"

This reverts commit 43e0e33da70a5f5c357d944ee9bc30db513b82bc.
crawler program

This manages the web crawler, database and frontier. it currently does
not inspect hyperlinks.