more informative error message
Merge branch 'diffbot-dan' of https://github.com/gigablast/open-source-search-engine into diffbot-dan
more informative error message
only apply Defect #2099 updates if it's a bulk job. I didn't see that variable yesterday
Defect #2122: If a crawl and there are no urlCrawlPattern or urlCrawlRegEx values, only return URLs from that domain
updated error message
do not download bulkjob urls in crawlbot.
just return a fake http reply.
however, do use crawl-delay throttling
logic. deduping is already turned off for
bulk jobs so it should be ok.
use SpiderRequest instead of isonsamedomain flag to determine whether to output data in CSV (Defect #2122)
Merge remote-tracking branch 'origin/diffbot' into diffbot-dan
add TODO for regex checking
when restarting a bulk job, copy bulkurls.txt to /tmp, and then transfer it back to the new collection folder
bigger buffer
Defect #2080: seed URLs duplicated
also include a timestamp field with an RFC 1123 formatted date
fix bug of gbmin, gbmax etc. not working.
floats were being rounded down to ints
in most cases it seems. so .9 -> 0 etc.
do not use certain other json fields
when computing checksum for deduping.
like stats, querystring, ...
Merge branch 'diffbot-dan' into diffbot-testing
fix critical title alloc/free bug
in title.cpp.
nomenclature changes to reduce collissions.
name collection 'qatest123' for doing smoke tests,
not 'test'.
implement something that works like mv, which tries rename first, and if that fails copies the bytes. rename doesn't work across devices