Modified code to use Postgresql database tables as queues to maintain the crawling history instead of java memory queues.
Added Dao classes for these change.
Added robotstxt-configuration.yml for strategy specific configuration
Added README.md file for project setup and dependencies.

Added database /tables creation scripts to db folder .