adding eclipse files to .gitignore
checking if the user-agent header contains the string defined in the crawler user-agents collection
variable typo: useAgent -> userAgent
only intercept GET requests
migrating to HttpClient 4