seenreq (npm): A library to test if a url(request) is crawled, usually used in a web crawler. Compatible with `request` and `node-crawler`
Keywords: nodejs|url seen test|reomve duplicate url|request normalize|crawler|duplicates-removed|post|request|spider|url