Remove one speaker and update talk speaker list accordingly