Remove some temporary matrices and factorize m2f and f2m