GitHub repository: nadar/crawler
Primary language: PHP
Description: A Website Crawler Implementation written in PHP. High extendible, Indexes PDFs and is very memory efficient.