Set swoop-font-size-change to nil.