Update internal state on unvoting