handle duplicate spidered media urls