Added pid unique key back so crawler can stop when detecting duplicate