not indexing dupes, only html checks, indexing title and uri