Only invert scaling when l=0; use initial multiplication/division direction