Only transform the scale if scale unequals 1