Move the permute flag the affine kernel.