added parallel for loop to remove duplicates. not sure if it works as intended.