the crawler now crawls more than one main page. Also changed the logging a bit