It should be possible to do this in place with Eigen3