Fix removed players not being taken off the surviving players list. See #180