[nemo-qml-plugin-contacts] Fix de-duplication error

Numbers that have no possibility of being duplicates are wrongly omitted from the result set.