checkpoint
new linkdb code compiling.
fixes for new link info code so it doesn't
bottleneck. got EFENCE_SIZE working so we
can use efence on large allocs only so we don't
go oom using it. might help finding some of
the out of bounds writing going on.
Merge branch 'diffbot' into diffbot-testing
Merge branch 'diffbot' into diffbot-testing
more fixes for new link info code.
fixes for query reindex/delete.
remove entry from linetable for linkinfo lookup
do not allow custom crawls to spider the web any more.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
add logic to save memory when streaming
over 200 results back. should fix oom when
streaming back hundreds of thousands of results.
Merge branch 'diffbot' into diffbot-testing
fix super fast and mem efficient search
results streaming code.
fix new result streaming logic some more
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
need to allow clients to not restrict to
seed domains.
fix core in "wait in line" logic
when getting link info in Linkdb.cpp.
fix core in new link text logic.
empty msg25 replies are ok if g_errno is set.
took out restrictDomain logic. now we always
only follow links on the same domain as the seed
UNLESS a url crawl pattern or a url crawl regex
was specified.
fix core from getting title of json object