Adding and removing ssh keys for users works now.