Re-adding validation to check if phantomjs is installed.
Avoiding loading images, improving error and log messages, handling
status errors properly not breaking the crawling process.
Improving all erros and logs with some statistics.

Adding --once and --stdout flag options for on-demand single-threaded
crawling.

Adding more filters to avoid images, compresses files and empty anchor
links.