Removed unnecessary memory transfer from CPU to CPU matrix

Found 15% performance improvement for matrix size around 3000