GitHub repository: marcostolosa/subcrawl
Description: SubCrawl is a modular framework for discovering open directories, identifying unique content through signatures and organizing the data with optional output modules, such as Elastic.