now ignores links that it has already processed.