Use log files to allow resume broken download with same or different starting url, and avoid repeated crawling.