crawler. make sure same url is never fetched twice
fix