microfrontier (npm): A web crawler frontier implementation in TypeScript backed by Redis. MicroFrontier is a scalable and distributed frontier implemented through Redis Queues.
Keywords: Web Crawler|Frontier|Search Engine|URL|Redis|Queue|Spider|Robots.txt|crawler|frontier|microservice|redis|robots-txt|spider