GitHub repository: valfirst/crawler-commons
Primary language: Java
Description: A set of reusable Java components that implement functionality common to any web crawler