Allow to remove the public key but not the user