fixing broken unit tests
optimization - not visiting dupe links