Prevent trying to remove users from the room if they're not already present

May help mitigate #852, guard against users that aren't present being removed from the list, possibly causing an exception.