Use separate polishing cloths or laps for each grit size and clean the workpiece thoroughly when switching between different grit sizes to prevent cross-contamination.