GitHub repository: kwahome/mirroring-web-crawler
Primary language: HTML
Description: A recursive, mirroring web crawler as a command-line tool that accepts a starting URL and a destination directory. The crawler will then download the page at the URL, save it in the destination directory, and then recursively proceed to any valid links in this page.