Merge branch 'diffbot'

Conflicts:

	Collectiondb.cpp
	Spider.cpp
	Spider.h
Merge branch 'diffbot'

Conflicts:

	Collectiondb.cpp
	Spider.cpp
	Spider.h
fix diffbot url bug
fix keep alive loop code to bail out if
fails to bind to socket as well as quick cores.
set safebuf::m_buf to null in destructor
do not let glibc do malloc checks, we do that.
fix add url and cached page getting
nomenclature. url filters -> spider scheduler
prevent msg20/tagdb lookup socket jam up.
throttle back max outstanding msg20s (summary generations)
based on used udp sockets.
fix core when thread fails to spawn.
time slice checkpoint
add connectips back. call them adminIps this time.
if your ip is on the list then you have admin
access. cookie tokens will come later/soon.
fix maxtocrawl limit thing
faster spiders checkpoint
spidercoll deletion fixes.
Merge branch 'master' into diffbot-slicing
checkpoint for faster spider code.
streaming results code checkpoint.
code checkpoint. time slicing, faster spider code
compiling. now needs debug.
spiders seem to be working somewhat now.