show lastSpidered time at end of json item.
it's a float so we should probably store it
as an int as well. we lose 128 seconds of resolution.
index numbers as integers too, not just floats
so we can sort by spider date without losing
128 seconds of resolution.
formatting fixes
remove bugg line
fix bug in gbminint.
fix core of page spiders
fix some cores. use olddoc contenthash
for msg13 call for EDOCUNCHANGED errors.
better logging msg when updating parm.
fix EDOCUNCHANGED logic.
EDOCUNCHANGED fixes for diffbot
use supplied mime if supplied when injecting
fix a core from bad return values
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
pass smoketest: TestOnlyProcessIfNew.testNotUpdatedContent
so diffbot reply will not be update in the index if it is unchanged
thereby keeping lastCrawlTimeUTC the same.
return json download as search results now.
all smokes have passed.
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
fi infinite loop core thing.
fix stack overflow
send single space to socket if not streaming
and search results still not ready after 10 seconds.
send it every 10 seconds to prevent client from closing socket.
sped up all downloads, json and csv, but not doing "fuzzy"
deduping of search results, but just deduping on page
content hash. added TcpSocket::m_numDestroys to ensure we
do not send heartbeat on a socket that was closed and
re-opened for another client.
do not do fuzzy deduping if &icc=1 (include cached copy)
is true for search results.