added fix for duplicate records returned by crawler