This is a branch of the Crawler4j project (https://code.google.com/p/crawler4j/), specifically I needed the option of adding authentication to URLs that require it.