com.github.crawler-commons:crawler-commons (maven): crawler-commons is a set of reusable Java components that implement functionality common to any web crawler. 
Keywords: java|library|open-source|robots-txt|sitemaps|web-crawler