There shouldn't be an action button for removing contacts from smart groups

Removing contacts from smart groups doesn't make sense, yet we have a button for this in the UI on the details page for a smart group.