crawlerx (npm): A powerful crawler support strategy to different url, the crawler can traverse all web page in a site recursively with certain deep. of course, you alse can do not use recursive crawel.
Keywords: crawl|spider|web|url