Don't switch the base layer if not necessary