added parm to reset proxy stats in table. erases
all our knowledge/stats for each proxy.
misnomer
update spider proxy logic.
update to spider proxy choose set logic
import patch from diffbot-testing
fix nytimes.com cookie/redir bug again.
add example for gbsortby:sitenuminlinks into syntax page
added sorting by site # inlinks/pop to menu for testing
remove graphix
tell diffbot to use gigablast host #0 as
its proxy and diffbot will in turn send
the request to one of the spider proxies in its list.
this way it can do its load balancing etc. algo.
Merge branch 'diffbot-testing' into testing
several fixes for floater proxy through squid proxy.
gb needs to act like squid for the rendering machines so
it can do crawl delay backoff and load balancing over the
floaters.
fix more bugs in squid proxy implementation.
force squid proxy stack to use floaters.
support the CONNECT for gb squid proxy
fix pageget.cpp
fixed ssl CONNECT reply from being truncated.
after receiving the CONNECT reply of ok we had
to reset the total bytes to read to 0 to avoid
using the old value and truncating the follow up reply
of the actual webserver response.
disable threads for disk and intersect/merge by default
for better performance since pthreads suck.
threads update for more warning msgs and
to save thread enabled status to gb.conf.