added duplicate remover preprocessing operator