yasserg/crawler4j — Issue #2: Number of crawled Pages varies
Author: Exeu
Created: 2012-01-13 10:14:01+00:00
State: open

Hi,

after an update to the new Crawler4J Version (3.0) the number of total Crawled Pages differs strongly for each run on the same site.

For example the first run results in (1076 Pages)
The second run directly after the first one results into (876 Pages)
The third one results in (971 Pages)

In the previous Versions there were no varies!

I've configured the crawler like the following:

CrawlConfig config = new CrawlConfig();

config.setCrawlStorageFolder(rootFolder);
config.setPolitenessDelay(400);
config.setMaxPagesToFetch(-1);
config.setMaxDepthOfCrawling(-1);
config.setResumableCrawling(false);
config.setIncludeHttpsPages(true);
config.setFollowRedirects(true);
config.setMaxConnectionsPerHost(200);
config.setMaxTotalConnections(200);

Is this an issue known to you?
Or could it be an usage failure?

Thanks