revise crawl.py for instant update of links