Copying matrices works, but we need to reconsider whether building a graph should add a reference or not