Fix duplicate memory usage matrices and handeling