change crawler to cache hashes and merge hashes before inserted into db