Remove player vote if they leave the lobby.