implemented duplicate reduction