add logging to the crawler and resolve #2