reverse boolean on authorized key remove

When we remove a key, we should only delete
the specific key and not every other key.