Set _spectral_unit only in with_spectral_unit