Don't upset vis state if not necessary

While this seems to be symptom of a larger issue, this change prevents the un-cool behavior seen in #1777.

Closes #1777