Replace submatrix access by per column access