yasserg/crawler4j — Issue #1: Threads Stop working
Author: camay015
Created: 2012-01-08 12:20:37+00:00
State: open

The crawler works with the web pages using in BasicCrawlcontroller
// controller.addSeed("http://www.ics.uci.edu/~yganjisa/");
// controller.addSeed("http://www.ics.uci.edu/~lopes/");
// controller.addSeed("http://www.ics.uci.edu/");

but NO other web pages appear to work. One example is below

Docid: 1
URL: http://www.unisa.edu.au/
Docid of parent page: 0
Text length: 2007
Html length: 14538
Number of outgoing links: 90
=============

10017 [Thread-1] INFO crawler.CrawlController - It looks like no thread is working, waiting for 10 seconds to make sure...
20023 [Thread-1] INFO crawler.CrawlController - No thread is working and no more URLs are in queue waiting for another 10 seconds to make sure...
30024 [Thread-1] INFO crawler.CrawlController - All of the crawlers are stopped. Finishing the process...
30025 [Thread-1] INFO crawler.CrawlController - Waiting for 10 seconds before final clean up...