Use Matrix4.copyInverse where appropriate