Imported code from Crawler4j
crawler-core readme