Normal users cannot un-assign people