[nemo-qml-plugin-contacts] Don't delete contacts held by the cache

Results in a double-delete abort.