GitHub repository: sebastian-nagel/crawler-commons
Primary language: Java
Description: A set of reusable Java components that implement functionality common to any web crawler