move recording of the finish-crawl of a page after all links on it have been processed, so if program crashes before this, next re-run can pick up this; also fixed a problem that caused double recording.