Updating dependencies and deprecated references
Fixed issue #127 where the port number was being ignored for robots.txt fetching.
Added generation of source and javadoc jars.
Removed some of the dependencies as they are already referenced by httpclient.
Minor code clean ups.
Fixed issues #156 #191 #192. Thanks to ilovesoup for reporting.
Adding meta refresh and location to outgoing links
Added support for handling content fetch and parsing issues.
Provide access to response headers in the visit method.
Fixed issue #169 and issue #136
Fixed issue #101
Fixed issue #184
Fixed issue #143
Upgrade to Java 7
Upgrade pom.xml to use Java 7
[maven-release-plugin] prepare release crawler4j-3.5
[maven-release-plugin] prepare for next development iteration
Fix Null pointer exception in Parser
added allows/ disallows directives for robot parser.
merged with upstream