Fixes #7
Upgrade BerkeleyDB JE to latest version

Fixes #10
Base crawler trap class.

Partial commit for issue #8
Partial commit for issue #8
Formatting only
Moved to commons 1.1.0 release
Minor POM adjustments
Refs #15. Base commit
Allowed maxDownloadSize to be set to -1 (unlimited)
Better logging: one message after starting all crawlers instead of one
per crawler.
Refs #17 PageFetcher error logging
Refs #18 Added utility class and logic in RobotsTxtServer and
PageFetcher
Refs #18 WebURL handles inline data URI
Fixes #19 Setting through CrawlConfig
Refs #19 : fixed CrawlConfig#toString()