filters duplicates and crawls every result now