Automating the SP duplicate elimination process