Make sure we don't set the mode if there is only a plot update