removed ide specific files; updated gitignore.
added support for custom-http-headers; bumped up the crawler4j version.
merged with upstream