Initial commit
Merge branch 'master' of https://github.com/Chaiavi/Crawler4j
Update README.md
Added log messages to PageFetcher

These logs were taken from:
ind9/crawler4j

153eb752bef5a57bd39016807423683ab22f3913
Fixes #21 - Updated pom.xml

Previous project referenceswere updated to current github project

Dependencies were updated to their latest

This commit closes #21
Changed from log4j implementation to slf4j API - Fixes #1

Removed any reference to log4j - Logging Implementations should be the
user's choice and not the library choice.

Instead I have inserted the slf4j API.

This commit closes #1
Added initial guidelines to the work of this prject
Added "How to use" & "Code examples" sections
Upgraded the lists
Upgrade all logging statements to use {} of slf4j #25

Slf4j optimizes string concatenation in logging statements by using {}
placeholder.

Upgrade all logging statements to use it.

Closes #25
Merge branch 'master' of https://github.com/Chaiavi/Crawler4j
Memory leakage in crawler4j caused by database environment #15

Closing the environment solves the leak.

Closes #15
Fixes #28 - Added binary content parsing

Now the crawler can really parse binary content.

Based on Tero Jankilla's code
Fixed some github-googlecode changes.
Fixed Issue #139

A URL with a backslash was rejected
Issue 276: Don't let a crawled URL to be dropped without proper logging.

Each URL should be logged somehow so the user will know it was crawled.
We shouldn't just drop URLs without logging them (warn / debug, depends on the case)

Fixes #276
Issue 278: Add hooks in the webcrawler for better error handling
Issue 239: Add an option to tweak the URL before processing the page

Added hooks to several errors where by default we log the error but allow the user to override those methods in order to do anything they want with those URLs (maybe list them and then try to process them later etc)
Added a method which can be overridden by the user which enables the user to tweak the url before fetching it
Issue 279:	TikaException is thrown while crawling several PDFs in a row

The problem was the wrong re-use of the metadata
Issue 251:	Fix a typo

Fix a typo in line #92. 'cuncurrent' -> 'concurrent'
Issue 284:	Cathing any exception and hidding the log.

Upgraded logging in webcrawler
Clarified a little statuscode 1005