if its a diffbot crawl/bulk job then do not use
linkdb to save disk space.