more informative error message
Merge branch 'diffbot-dan' of https://github.com/gigablast/open-source-search-engine into diffbot-dan
more informative error message
only apply Defect #2099 updates if it's a bulk job. I didn't see that variable yesterday
Defect #2122: If a crawl and there are no urlCrawlPattern or urlCrawlRegEx values, only return URLs from that domain
updated error message
use SpiderRequest instead of isonsamedomain flag to determine whether to output data in CSV (Defect #2122)
Merge remote-tracking branch 'origin/diffbot' into diffbot-dan
add TODO for regex checking
when restarting a bulk job, copy bulkurls.txt to /tmp, and then transfer it back to the new collection folder
bigger buffer
Defect #2080: seed URLs duplicated
also include a timestamp field with an RFC 1123 formatted date
Merge branch 'diffbot-dan' into diffbot-testing