disregard CONNECT requests for now
Merge branch 'diffbot' of github.com:gigablast/open-source-search-engine into diffbot

Conflicts:
	HttpServer.cpp
fix core
Merge branch 'diffbot' into diffbot-testing
instructions update
Merge pull request #32 from emmanuelcharon/diffbot-testing

rename diffbotHopcount to maxHops
fix a couple of cores.
reduce memory usage significantly by not
pre-allocating some per-collection hashtables.