GitHub repository: scrapinghub/aduana
Primary language: C
Description: Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).