added pipeline to drop dupes