GitHub repository: jstrosch/subcrawl
Primary language: Python
Description: SubCrawl is a modular framework for discovering open directories, identifying unique content through signatures and organizing the data with optional output modules, such as Elastic.