Eliminate duplicate crawls, store history in db if received