Don't remove spellmod CASTING_TIME, already handeled at spell::cast so we skip there.