Added reverse references to Redis so that we now store an array of pids for each url so we can easily identify duplicates