Update memory allocate matrix, only update self in leaf node