[maven-release-plugin] prepare release crawler4j-3.4
[maven-release-plugin] prepare for next development iteration
Implement a registry to dynamically add different content types to the parser.
added hidden file
Merge pull request #1 from rmetzger/robert_master

added hidden file
Updating dependencies and deprecated references
Fixed issue #127 where the port number was being ignored for robots.txt fetching.
Added generation of source and javadoc jars.
Removed some of the dependencies as they are already referenced by httpclient.
Minor code clean ups.
Fixed issues #156 #191 #192. Thanks to ilovesoup for reporting.
Adding meta refresh and location to outgoing links
Added support for handling content fetch and parsing issues.
Provide access to response headers in the visit method.
Fixed issue #169 and issue #136
Fixed issue #101
Fixed issue #184
Fixed issue #143
Upgrade to Java 7
Upgrade pom.xml to use Java 7