use new htmlunit version and backtrack httpclient version for incompatibility

htmlunit accesses a private static httpclient method using reflection and it was removed in 4.2.4, breaking htmlunit...
Remove unused profiles and add note about test.log4j.properties
remove old configuration files and disable hardcoded hostname for now
bump versions and add dependencies as necessary
do not automatically blacklist clients to reduce memory usage
cleanup
fix bug with rdf json
reduce logging
update plugin versions
bump some dependency versions